[galaxy-dev] Release of Galaxy 22.05

[Marius van den Beek]

Dear Community,
We are pleased to announce the release of Galaxy 22.05.


   - Developer and admin release announcement
   .
   - User release announcement
   
   .

A few release highlights are:
The New History is Here

After years of effort by the development team and community contributors,
the new history is finally here. It's a big improvement over the old
history interface, both technically and in terms of user experience, but it
might take some getting used to. It has a lot of great new features like:

   - A quick switcher, letting you switch between recent histories
   - Easily find dataset inputs
   - Improved history search interface

Check it out now!
Storage Dashboard (Beta)

Have you ever been over your quota? 250 GB sure goes fast! Now with the new
Storage Dashboard you can get a quick overview of ways to help clean up
deleted datasets and recover some of that quota allocation. This is a beta
version of this interface, let us know if you have any issues. In future
versions, it will be expanded to help you understand how your allocation is
being used, and which histories and datasets should be cleaned up first.
Bulk History Operations

The new history allows for improved bulk operations, selecting dozens or
hundreds of datasets to retag or change the database key automatically, a
process that used to require using collections or doing them one by one.
Deferred dataset resolution enables running workflows and tools without
importing data into Galaxy first

It is now possible to defer dataset resolution for datasets imported via
URLs and the "Choose remote files" dialog. This means that the deferred
dataset will only be downloaded as it is needed during job execution.
Deferred datasets will not count towards your storage quota, as the data is
not stored by Galaxy. To enable deferred dataset resolution click on
"Settings" in the upload dialog.
Workflow Improvements

You can now see all invocations of a specific workflow, across all of your
histories.

This is especially useful if you run a single workflow across multiple
datasets, and keep them nicely separated in multiple histories. You can see
all of the results in a single centralised location.

Additional workflow improvements have been made, e.g. numbering of steps in
a workflow. Improvements have been made to the internal representation of
the workflow that should make the saved .ga and .gxwf.yml files easier to
compare across changes.

Workflows can also now be zoomed in and out via scrolling, this interface
should be more familiar for anyone who has used Google Maps:
Saved Rules

Do you use the Rule Based Uploader (RBU)? (If not, learn how today!
)
The RBU now has a very convenient recently used rule listing, sorted by how
recently they were used. You can hover over each entry to also list a
preview of what steps were done in that RBU invocation.
Scratchbook Upgraded

The scratchbook has been updated to a new implementation. Now your views of
datasets can overlap, you have much more freedom in the size of each
window, and you can minimize datasets like minimizing a window to the
taskbar in Windows.
OpenAPI Docs

The Galaxy API is going through an upgrade to FastAPI, which allows us to
generate OpenAPI documentation . If you're
an API consumer, either via BioBlend or via another system, you can use
this to see all of the APIs available from Galaxy and try them out live.
Release Notes

Please see the full release notes
 for
a lot more details and instructions for upgrading your Galaxy installation.

Thanks for using Galaxy!
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Release of Galaxy 22.01

[Marius van den Beek]

Dear Community,

The Galaxy Committers team is pleased to announce the release of Galaxy
22.01.

   - Developer and admin release announcement
   .
   - User release announcement
   
   .

A few release highlights are:
New Colour Selector

Do you use tools which require colour inputs, like Circos? Previously, we
had a very restricted colour input which gave you a very limited palette.
Now, you have complete freedom of choice with the modern colour selector.
Improved File Export

If you've been exporting files from Galaxy lately, you've probably seen the
amazing new remote file source export, which allows you to export files to
FTP, Dropbox, and other locations.
Improved File Uploads

Previously, Galaxy servers used a variety of methods to let you upload
large files easily, including some servers which required FTP for large
files. We have replaced this with a new upload method, which will be
enabled on all usegalaxy servers soon. This will make file uploads
significantly smoother, and will be more tolerant of network failures and
interruptions! You do not need to make any changes.
Beta History: Collection Improvements

If you've been trying out the beta history (which will be the default
history in the next release!), it has been updated to indicate whether
collections are homogeneous or heterogeneous. This will help you see more
easily if you've accidentally included an incorrect dataset.
Galaxy starts as a FastAPI application by default

Starting Galaxy via run.sh will use the new gravity process manager
. The new configuration uses
Gunicorn  and FastAPI  to
drive Galaxy's web process, and starts job handler and Celery
 processes automatically. For more details
and instructions, please consult the Migrating to Gunicorn documentation
.
User Preferences can be encrypted in Galaxy Vault

Galaxy can now be configured to store secrets in an external vault, which
is useful for secure handling and centralization of secrets management. In
particular, information fields in the "Manage information" section of the
user profile, such as dropbox keys, can be configured to be encrypted at
rest in a vault (Hashicorp, Custos or database) instead of being stored as
plain text in the user preferences table. For detailed information on
configuration, refer to the vault section

of
the admin documentation.
Release Notes

Please see the full release notes
 for
a lot more details and instructions for upgrading your Galaxy installation.

Thanks for using Galaxy!
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Re: Cannot update tool on toolshed

[Marius van den Beek]

Hi Petr,

Have you tried using `planemo lint` to check that your tools are
structurally correct and `planemo shed_update` (
https://planemo.readthedocs.io/en/latest/publishing.html) to upload
repositories ?
I'm afraid the upload interface of the tool shed currently only works if
there are no issues in the tools or the tools' metadata,
and most of the community has adapted to using continuous integration to
test and upload tools.
We have a template repository that you could adopt for this at
https://github.com/galaxyproject/galaxy-tool-repository-template.

Best,
Marius

On Mon, 24 Jan 2022 at 08:02,  wrote:

> I have repository
> https://toolshed.g2.bx.psu.edu/repository/browse_repository?id=691d87250e824103
> When I try to update repository by uploading either tarball or single
> file, nothing happen - I got a blank page which never updates. When I
> explored the content of this repository it look like the content was
> updated partially - Toolshed still shows the revision number 0 but  content
> of files visible through toolshed is updated. However, linked mercurial
> repository at
> https://petr-no...@toolshed.g2.bx.psu.edu/repos/petr-novak/repeat_annotation_pipeline3
> does not have this update. So the content visible through tooshed interface
> different from content of hg repository.
> This is not a first time it happened, I have also other repositories which
> cannot be updated. Is it a bug or am I doing something wrong?
>
> Petr
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Release of Galaxy 21.09

[Marius van den Beek]

Dear Community,

The Galaxy Committers team is pleased to announce the release of Galaxy
21.09.

   - Developer and admin release announcement
   
   - User release announcement
   
   .

A few release highlights are:
Tool Panel Views

We now have "Tool Panel Views"! These are different views into the same
toolbox and might help make it easier to find the tools you want. In the
future, there are plans to create user-customisable toolboxes, but until
then go explore the new EDAM Ontology-based toolboxes which organise tools
around scientific areas or processes. For example, a category like
"Filtering" might have tools like "select lines" or "filter bam by
quality", both doing the same process of filtering, despite their different
file types and formats.

Unfortunately, not all tools are fully annotated yet, and while we also
pull terms from bio.tools, this still doesn't get us full coverage and many
tools will still appear under a large section "Uncategorized". Hopefully,
this will improve as users and tool developers annotate these tools with
the appropriate terms that help everyone find them.
Collections (Beta History Updates)

Did you ever create *a collection with the wrong dbkey or datatype*? Well,
in the Beta History you can now change collections' datatypes and dbkeys!
This should save a lot of time for everyone working with large datasets (Pull
Request 11799 ).

Additionally, did a *collection ever fail* for you? And you wondered why?
Now you can find out with the view details button for collections (Pull
Request 12261 )!
Importing Data

Selecting datasets from the Remote Files view has gotten easier! Now, you
can select folders and files and import at once all of the datasets
recursively in those folders. Previously, you could only select files
within a folder, so this is a huge improvement in the usability of such a
key new feature of Galaxy (Pull Request 12310
). But with great power
comes great responsibility, be careful not to import the entirety of NCBI!
Reports

Report components used to automatically arrange themselves in a smart way,
but due to some issues with the floating of components, these will now be
full width until we can figure out a good way to provide a similar
feature (Pull
Request 12085 ).

Enhancements for Working with Remote Data and Distributed Computing
Resources

Many fixes and enhancements were made to improve how Galaxy can import and
write to remote data using the configured file source plugins. In addition
to the History Export functionality, which can write to remote locations,
we have added an Export datasets Tool for exporting individual Datasets and
Dataset collections to configured remote locations. The Tool will
automatically maintain the name and structure of Datasets and Dataset
Collections. We have added the possibility to import entire folders of
remote data and made the data selection dialog more intuitive. We also
added numerous improvements and fixes to the extended metadata collection
strategy and Pulsar , so that many
more tools are able to run in Pulsar
 and write data back to storage
without passing through Galaxy first.
Migration to FastAPI and Extended Documentation of API Routes

We have modernized and migrated many more API routes to FastAPI and have
extended the documentation and validation of these routes.
Migration to SQLAlchemy 1.4 and Declarative Mapping

Galaxy is now using SQLAlchemy  version 1.4,
which is a prerequisite for asynchronously interacting with the database.
We also modernized the way our database models are defined to the more
commonly used Declarative Mapping approach, which is more concise and
better documented.
Modernization of Tool Form Interface

The tool form interface has been almost completely migrated from Backbone
 to Vue . This improves the
reactivity of parameter validation and enables the migration of tool
parameters to Vue, which will also allow us to add many more types of tool
parameters. We further migrated the tool form variant for the Workflow
Editor, the Workflow run form and the Show Dataset Parameter page to Vue.
New User Welcome Page

We have added a page that new users will be directed to after creating a
new Galaxy account.
Release Notes

Please see the full release notes
 for
a lot more details and instructions for upgrading your Galaxy installation.

Thanks for using Galaxy!

[galaxy-dev] Re: Galaxy install problems

[Marius van den Beek]

Hi Luc,

you need to capitalize the first letter of slurm in your config:

pulsar_yaml_config:
  conda_auto_init: True
  conda_auto_install: True
  staging_directory: "{{ pulsar_staging_dir }}"
  persistence_directory: "{{ pulsar_persistence_dir }}"
  tool_dependency_dir: "{{ pulsar_dependencies_dir }}"
  # The following are the settings for the pulsar server to contact the
message queue with related timeouts etc.
  message_queue_url: "pyamqp://galaxy_au:{{ rabbitmq_password_galaxy_au
}}@{{ galaxy_server_url }}:5671//pulsar/galaxy_au?ssl=1"
  managers:
_default_:
  type: queued_cli
  job_plugin: Slurm
  native_specification: "-p batch --tasks=1 --cpus-per-task=2
--mem-per-cpu=1000 -t 10:00"
  min_polling_interval: 0.5
  amqp_publish_retry: True
  amqp_publish_retry_max_retries: 5
  amqp_publish_retry_interval_start: 10
  amqp_publish_retry_interval_step: 10
  amqp_publish_retry_interval_max: 60

Best,
Marius

On Wed, 30 Jun 2021 at 18:35, Luc Cornet  wrote:

> Re,
>
> We shot down pulsar, run the playbook again, restart pulsar:
> We I try to launch an analysis, I have this error for the job (still CLI):
>
> Failed to find job_plugin of type slurm, available types include ['LSF',
> 'Slurm', 'SlurmTorque', 'Torque']
>
> Luc,
>
> 
> Luc Cornet, PhD
> Bio-informatician
> Mycology and Aerobiology
> Sciensano
>
> - Mail original -
> De: "Marius van den Beek" 
> À: "Luc Cornet" 
> Cc: "HelpGalaxy" , "Baurain Denis" <
> denis.baur...@uliege.be>, "Pierre Becker" ,
> "Colignon David" 
> Envoyé: Mercredi 30 Juin 2021 17:24:13
> Objet: [galaxy-dev] Re: Galaxy install problems
>
> That should be `systemctl restart pulsar.service`
>
> Marius
>
> On Wed, 30 Jun 2021 at 17:05, Luc Cornet < [ mailto:luc.cor...@uliege.be
> | luc.cor...@uliege.be ] > wrote:
>
>
> Re,
>
> This is the log of pulsar.
>
> As pointed by Keith Superman, the pulsar don't restart since the step is
> skipped.
> How can I force pulsar to restart ?
>
> Thanks,
> Luc
>
> 
> Luc Cornet, PhD
> Bio-informatician
> Mycology and Aerobiology
> Sciensano
>
> - Mail original -
> De: "Marius van den Beek" < [ mailto:m.vandenb...@gmail.com |
> m.vandenb...@gmail.com ] >
> À: "Luc Cornet" < [ mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ]
> >
> Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org |
> galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto:
> denis.baur...@uliege.be | denis.baur...@uliege.be ] >, "Pierre Becker" <
> [ mailto:pierre.bec...@sciensano.be | pierre.bec...@sciensano.be ] >,
> "Colignon David" < [ mailto:david.colig...@uliege.be |
> david.colig...@uliege.be ] >
> Envoyé: Mercredi 30 Juin 2021 16:44:50
> Objet: Re: [galaxy-dev] Re: Galaxy install problems
>
> If you've set up pulsar to start as a systemd service you should get the
> logs with journalctl.
> I assume that's `journalctl -u pulsar.service`.
> For a quickstart in journalctl check out
> [ https://www.linode.com/docs/guides/how-to-use-journalctl/ |
> https://www.linode.com/docs/guides/how-to-use-journalctl/ ]
>
>
> On Wed, 30 Jun 2021 at 16:38, Luc Cornet < [ mailto:luc.cor...@uliege.be
> | luc.cor...@uliege.be ] > wrote:
>
> > OK,
> >
> > Sure but can you please tell me how to get these pulsar application logs
> ?
> >
> > Thanks
> > Luc
> >
> > 
> > Luc Cornet, PhD
> > Bio-informatician
> > Mycology and Aerobiology
> > Sciensano
> >
> > - Mail original -
> > De: "Marius van den Beek" < [ mailto:m.vandenb...@gmail.com |
> m.vandenb...@gmail.com ] >
> > À: "Luc Cornet" < [ mailto:luc.cor...@uliege.be | luc.cor...@uliege.be
> ] >
> > Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org |
> galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" <
> > [ mailto:denis.baur...@uliege.be | denis.baur...@uliege.be ] >, "Pierre
> Becker" < [ mailto:pierre.bec...@sciensano.be | pierre.bec...@sciensano.be
> ] >,
> > "Colignon David" < [ mailto:david.colig...@uliege.be |
> david.colig...@uliege.be ] >
> > Envoyé: Mercredi 30 Juin 2021 16:32:16
> > Objet: [galaxy-dev] Re: Galaxy install problems
> >
> > Thanks, but these are not the pulsar application logs. Can you provide
> > these please

[galaxy-dev] Re: Galaxy install problems

[Marius van den Beek]

That should be `systemctl restart pulsar.service`

Marius

On Wed, 30 Jun 2021 at 17:05, Luc Cornet  wrote:

> Re,
>
> This is the log of pulsar.
>
> As pointed by Keith Superman, the pulsar don't restart since the step is
> skipped.
> How can I force pulsar to restart ?
>
> Thanks,
> Luc
>
> 
> Luc Cornet, PhD
> Bio-informatician
> Mycology and Aerobiology
> Sciensano
>
> - Mail original -
> De: "Marius van den Beek" 
> À: "Luc Cornet" 
> Cc: "HelpGalaxy" , "Baurain Denis" <
> denis.baur...@uliege.be>, "Pierre Becker" ,
> "Colignon David" 
> Envoyé: Mercredi 30 Juin 2021 16:44:50
> Objet: Re: [galaxy-dev] Re: Galaxy install problems
>
> If you've set up pulsar to start as a systemd service you should get the
> logs with journalctl.
> I assume that's `journalctl -u pulsar.service`.
> For a quickstart in journalctl check out
> https://www.linode.com/docs/guides/how-to-use-journalctl/
>
>
> On Wed, 30 Jun 2021 at 16:38, Luc Cornet  wrote:
>
> > OK,
> >
> > Sure but can you please tell me how to get these pulsar application logs
> ?
> >
> > Thanks
> > Luc
> >
> > 
> > Luc Cornet, PhD
> > Bio-informatician
> > Mycology and Aerobiology
> > Sciensano
> >
> > - Mail original -
> > De: "Marius van den Beek" 
> > À: "Luc Cornet" 
> > Cc: "HelpGalaxy" , "Baurain Denis" <
> > denis.baur...@uliege.be>, "Pierre Becker" ,
> > "Colignon David" 
> > Envoyé: Mercredi 30 Juin 2021 16:32:16
> > Objet: [galaxy-dev] Re: Galaxy install problems
> >
> > Thanks, but these are not the pulsar application logs. Can you provide
> > these please ?
> >
> > On Wed, 30 Jun 2021 at 16:30, Luc Cornet < [ mailto:luc.cor...@uliege.be
> > | luc.cor...@uliege.be ] > wrote:
> >
> >
> > Dear Marius,
> >
> > Many thank for your feedback.
> >
> > I join to this email: the playbook, the pulsarservers.yml file and the
> log
> > of pulsar playbook.
> >
> > CLI plugin is for us the best solution since we have nothing to maintain.
> > DRMAA is not actively developed for slurm, correct ?
> >
> >
> > In the playbook, we use systemd which I think should restart pulsar but
> It
> > might not be the case:
> >
> > TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start
> > service]
> >
> 
> >
> > ok: [HPC]
> >
> > RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler]
> >
> *
> >
> > skipping: [HPC]
> >
> > Currently, we never used DRMAA. The job were executed immediately on the
> > cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to
> > activate CLI:
> > managers:
> > _default_:
> > type: queued_cli
> > job_plugin: slurm
> > native_specification: "-p batch --tasks=1 --cpus-per-task=2
> > --mem-per-cpu=1000 -t 10:00"
> > min_polling_interval: 0.5
> > amqp_publish_retry: True
> > amqp_publish_retry_max_retries: 5
> > amqp_publish_retry_interval_start: 10
> > amqp_publish_retry_interval_step: 10
> > amqp_publish_retry_interval_max: 60
> >
> >
> > Thanks for your help,
> > Luc
> >
> >
> > 
> > Luc Cornet, PhD
> > Bio-informatician
> > Mycology and Aerobiology
> > Sciensano
> >
> > - Mail original -
> > De: "Marius van den Beek" < [ mailto:m.vandenb...@gmail.com |
> > m.vandenb...@gmail.com ] >
> > À: "Luc Cornet" < [ mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ]
> > >
> > Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org |
> > galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto:
> > denis.baur...@uliege.be | denis.baur...@uliege.be ] >, "Pierre Becker" <
> > [ mailto:pierre.bec...@sciensano.be | pierre.bec...@sciensano.be ] >,
> > "Colignon David" < [ mailto:david.colig...@uliege.be |
> > david.colig...@uliege.be ] >
> > Envoyé: Mercredi 30 Juin 2021 16:02:04
> > Objet: [galaxy-dev] Re: Galaxy install problems
> >
> > Hi Luc,
> >
> > I'm sorry to hear that you're struggling to set up Galaxy to your liking.
> > Let me start b

[galaxy-dev] Re: Galaxy install problems

[Marius van den Beek]

If you've set up pulsar to start as a systemd service you should get the
logs with journalctl.
I assume that's `journalctl -u pulsar.service`.
For a quickstart in journalctl check out
https://www.linode.com/docs/guides/how-to-use-journalctl/


On Wed, 30 Jun 2021 at 16:38, Luc Cornet  wrote:

> OK,
>
> Sure but can you please tell me how to get these pulsar application logs ?
>
> Thanks
> Luc
>
> 
> Luc Cornet, PhD
> Bio-informatician
> Mycology and Aerobiology
> Sciensano
>
> ----- Mail original -
> De: "Marius van den Beek" 
> À: "Luc Cornet" 
> Cc: "HelpGalaxy" , "Baurain Denis" <
> denis.baur...@uliege.be>, "Pierre Becker" ,
> "Colignon David" 
> Envoyé: Mercredi 30 Juin 2021 16:32:16
> Objet: [galaxy-dev] Re: Galaxy install problems
>
> Thanks, but these are not the pulsar application logs. Can you provide
> these please ?
>
> On Wed, 30 Jun 2021 at 16:30, Luc Cornet < [ mailto:luc.cor...@uliege.be
> | luc.cor...@uliege.be ] > wrote:
>
>
> Dear Marius,
>
> Many thank for your feedback.
>
> I join to this email: the playbook, the pulsarservers.yml file and the log
> of pulsar playbook.
>
> CLI plugin is for us the best solution since we have nothing to maintain.
> DRMAA is not actively developed for slurm, correct ?
>
>
> In the playbook, we use systemd which I think should restart pulsar but It
> might not be the case:
>
> TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start
> service]
> 
>
> ok: [HPC]
>
> RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler]
> *
>
> skipping: [HPC]
>
> Currently, we never used DRMAA. The job were executed immediately on the
> cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to
> activate CLI:
> managers:
> _default_:
> type: queued_cli
> job_plugin: slurm
> native_specification: "-p batch --tasks=1 --cpus-per-task=2
> --mem-per-cpu=1000 -t 10:00"
> min_polling_interval: 0.5
> amqp_publish_retry: True
> amqp_publish_retry_max_retries: 5
> amqp_publish_retry_interval_start: 10
> amqp_publish_retry_interval_step: 10
> amqp_publish_retry_interval_max: 60
>
>
> Thanks for your help,
> Luc
>
>
> 
> Luc Cornet, PhD
> Bio-informatician
> Mycology and Aerobiology
> Sciensano
>
> - Mail original -
> De: "Marius van den Beek" < [ mailto:m.vandenb...@gmail.com |
> m.vandenb...@gmail.com ] >
> À: "Luc Cornet" < [ mailto:luc.cor...@uliege.be | luc.cor...@uliege.be ]
> >
> Cc: "HelpGalaxy" < [ mailto:galaxy-dev@lists.galaxyproject.org |
> galaxy-dev@lists.galaxyproject.org ] >, "Baurain Denis" < [ mailto:
> denis.baur...@uliege.be | denis.baur...@uliege.be ] >, "Pierre Becker" <
> [ mailto:pierre.bec...@sciensano.be | pierre.bec...@sciensano.be ] >,
> "Colignon David" < [ mailto:david.colig...@uliege.be |
> david.colig...@uliege.be ] >
> Envoyé: Mercredi 30 Juin 2021 16:02:04
> Objet: [galaxy-dev] Re: Galaxy install problems
>
> Hi Luc,
>
> I'm sorry to hear that you're struggling to set up Galaxy to your liking.
> Let me start by pointing out that [ [ http://usegalaxy.org/ |
> http://usegalaxy.org/ ] | [ http://usegalaxy.org/ | usegalaxy.org ] ]
> uses slurm with DRMAA, this is certainly going to be more performant and
> reliable than the CLI plugin.
> There is little maintenance necessary, so maybe that is why activity on
> slurm-drmaa is low (See also [ [ https://github.com/natefoo/slurm-drmaa |
> https://github.com/natefoo/slurm-drmaa ] | [
> https://github.com/natefoo/slurm-drmaa |
> https://github.com/natefoo/slurm-drmaa ] ] ).
> I would be curious to know how you came to the conclusion that there is
> some incompatibility between DRMAA and slurm
> Note that one of the setups we teach during the training submits via DRMAA
> to slurm.
>
> Then I'd like to point out that there are a huge variety of different ways
> in which you can configure Galaxy and the job submission.
> We teach the most common ones during the training week, with the aim that
> you understand how these things work together,
> as well as giving you a handle on how you can manage these different
> settings and services using a configuration management system.
> We cannot tailor a solution to your infrastructure during this week.
>
> About your problem specifically, I had asked this on gitter before:
>
> > Did you restart pulsar af

[galaxy-dev] Re: Galaxy install problems

[Marius van den Beek]

Thanks, but these are not the pulsar application logs. Can you provide
these please ?

On Wed, 30 Jun 2021 at 16:30, Luc Cornet  wrote:

> Dear Marius,
>
> Many thank for your feedback.
>
> I join to this email: the playbook, the pulsarservers.yml file and the log
> of pulsar playbook.
>
> CLI plugin is for us the best solution since we have nothing to maintain.
> DRMAA is not actively developed for slurm, correct ?
>
>
> In the playbook, we use systemd which I think should restart pulsar but It
> might not be the case:
>
> TASK [galaxyproject.pulsar : systemd daemon-reload and enable/start
> service]
> 
> ok: [HPC]
>
> RUNNING HANDLER [galaxyproject.pulsar : default restart pulsar handler]
> *
> skipping: [HPC]
>
> Currently, we never used DRMAA. The job were executed immediately on the
> cluster with CLI or DRMAA. We had this part in pulsarservers.yml, to
> activate CLI:
>   managers:
> _default_:
>   type: queued_cli
>   job_plugin: slurm
>   native_specification: "-p batch --tasks=1 --cpus-per-task=2
> --mem-per-cpu=1000 -t 10:00"
>   min_polling_interval: 0.5
>   amqp_publish_retry: True
>   amqp_publish_retry_max_retries: 5
>   amqp_publish_retry_interval_start: 10
>   amqp_publish_retry_interval_step: 10
>   amqp_publish_retry_interval_max: 60
>
>
> Thanks for your help,
> Luc
>
>
> ----
> Luc Cornet, PhD
> Bio-informatician
> Mycology and Aerobiology
> Sciensano
>
> - Mail original -
> De: "Marius van den Beek" 
> À: "Luc Cornet" 
> Cc: "HelpGalaxy" , "Baurain Denis" <
> denis.baur...@uliege.be>, "Pierre Becker" ,
> "Colignon David" 
> Envoyé: Mercredi 30 Juin 2021 16:02:04
> Objet: [galaxy-dev] Re: Galaxy install problems
>
> Hi Luc,
>
> I'm sorry to hear that you're struggling to set up Galaxy to your liking.
> Let me start by pointing out that [ http://usegalaxy.org/ | usegalaxy.org
> ] uses slurm with DRMAA, this is certainly going to be more performant and
> reliable than the CLI plugin.
> There is little maintenance necessary, so maybe that is why activity on
> slurm-drmaa is low (See also [ https://github.com/natefoo/slurm-drmaa |
> https://github.com/natefoo/slurm-drmaa ] ).
> I would be curious to know how you came to the conclusion that there is
> some incompatibility between DRMAA and slurm
> Note that one of the setups we teach during the training submits via DRMAA
> to slurm.
>
> Then I'd like to point out that there are a huge variety of different ways
> in which you can configure Galaxy and the job submission.
> We teach the most common ones during the training week, with the aim that
> you understand how these things work together,
> as well as giving you a handle on how you can manage these different
> settings and services using a configuration management system.
> We cannot tailor a solution to your infrastructure during this week.
>
> About your problem specifically, I had asked this on gitter before:
>
> > Did you restart pulsar after rolling out the new config ?
>
> to which you've answered that you re-ran the playbook, but that's not a
> sufficient answer.
>
> Every playbook is different, and we cannot know if this includes a
> restarter service for pulsar.
> Also please don't assume that everyone that could potentially help you
> knows ansible and the playbooks that are being taught intimately,
> and in what ways you have customized your playbook.
> It is much more helpful to write up the relevant settings you've changed
> and the logs that go with it.
>
> You've also been asked to provide logs of the restart, which as far as I
> can tell you haven't provided.
> You had mentioned on gitter that pulsar continues to use DRMAA to submit
> jobs, so you'll
> want to double check whether you've really restarted pulsar after the
> config changes,
> and look at the startup logs for pulsar, and find out how it is possible
> for pulsar to submit jobs
> via drmaa if it is not set up to do so.
>
> Best,
> Marius
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Re: Galaxy install problems

[Marius van den Beek]

Hi Luc,

I'm sorry to hear that you're struggling to set up Galaxy to your liking.
Let me start by pointing out that usegalaxy.org uses slurm with DRMAA, this
is certainly going to be more performant and reliable than the CLI plugin.
There is little maintenance necessary, so maybe that is why activity on
slurm-drmaa is low (See also https://github.com/natefoo/slurm-drmaa).
I would be curious to know how you came to the conclusion that there is
some incompatibility between DRMAA and slurm
Note that one of the setups we teach during the training submits via DRMAA
to slurm.

Then I'd like to point out that there are a huge variety of different ways
in which you can configure Galaxy and the job submission.
We teach the most common ones during the training week, with the aim that
you understand how these things work together,
as well as giving you a handle on how you can manage these different
settings and services using a configuration management system.
We cannot tailor a solution to your infrastructure during this week.

About your problem specifically, I had asked this on gitter before:

> Did you restart pulsar after rolling out the new config ?

to which you've answered that you re-ran the playbook, but that's not a
sufficient answer.

Every playbook is different, and we cannot know if this includes a
restarter service for pulsar.
Also please don't assume that everyone that could potentially help you
knows ansible and the playbooks that are being taught intimately,
and in what ways you have customized your playbook.
It is much more helpful to write up the relevant settings you've changed
and the logs that go with it.

You've also been asked to provide logs of the restart, which as far as I
can tell you haven't provided.
You had mentioned on gitter that pulsar continues to use DRMAA to submit
jobs, so you'll
want to double check whether you've really restarted pulsar after the
config changes,
and look at the startup logs for pulsar, and find out how it is possible
for pulsar to submit jobs
via drmaa if it is not set up to do so.

Best,
Marius
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Release of Galaxy 21.05

[Marius van den Beek]

Dear Community,

The Galaxy Committers team is pleased to announce the release of Galaxy
21.05.

The release announcement for developers and admins can be found at
https://docs.galaxyproject.org/en/master/releases/21.05_announce.html and
user facing release notes are at
https://docs.galaxyproject.org/en/master/releases/21.05_announce_user.html.

A few release highlights are:

¡Galaxy, ahora en español!

Thanks to Wendi Bacon, the Spanish language translation of Galaxy has been
finalised and merged, so if you prefer to use Galaxy in Spanish, now you
can!
This update will be part of an ongoing project from Spanish speakers within
the
Galaxy community to keep the Galaxy interface localisation up to date,
and to produce some Spanish language training materials in the GTN.

Bugfixes and Stability
-
This release of Galaxy features fewer user-facing changes, as a huge amount
of
developer time went into making this a maintenance release with better
testing,
better stability, and more bugfixes.
But watch out, this is all in preparation for the next release of Galaxy,
21.09,
which will have some of the biggest changes in years!

New development stack

Galaxy release 21.09 will ship with a new web framework (fastAPI
),
Celery  task queue and
process management using Circus .
You can preview this new stack now by running *APP_WEBSERVER=dev ./run.sh*.

Celery for background tasks
--
Galaxy can now run certain tasks in the background. The Celery workers are
currently not required, but if activated can perform certain long-running
tasks,
such as creating history export archives. Celery tasks will bridge the gap
between
rapid requests that can be handled during a web request and jobs that
require extensive
and relatively slow setup.

More robust selection of job handlers
-
Job throughput can be increased by starting Galaxy with multiple external
job
handler processes. Jobs were traditionally assigned to a job handler process
by the web handler or workflow handler process that created the job. Since
Release 19.01 Galaxy has supported additional mechanisms that use database
serialization techniques to let job handlers assign processes to themselves.
This mechanism is more robust and doesn't require that all job handler
processes be alive and known by the web handler process. Galaxy now
determines
the best method for assigning jobs based on the database in use, if the
assignment
method is not set explicitly. Older job assignment methods will be removed
in Galaxy
release 21.09. For more details see the Job Handler Assignment Methods
section

of the Galaxy documentation.

Check out the full release notes
 for
a lot more details, there are many more
enhancements, new visualizations and bugfixes as well as instructions for
upgrading your Galaxy installation.

Thanks for using Galaxy!
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Release of Galaxy 21.01

[Marius van den Beek]

Dear Community,

The Galaxy Committers team is pleased to announce the release of Galaxy
21.01.

The release announcement for developers and admins can be found at
https://docs.galaxyproject.org/en/master/releases/21.01_announce.html and
user facing release notes are at
https://docs.galaxyproject.org/en/master/releases/21.01_announce_user.html.

A few release highlights are:

Workflows: Best Practices, Reports, Invocations

Workflows have seen huge improvements in this release! The workflow report
editor
is easier than ever to use, providing you with a list of common report
components, interactive interfaces for embedding them in your reports, and a
new workflow invocation tracker. You can now embed visualizations directly
in
your workflow reports making summarizing your analyses easier than ever.
And,
once your reports are produced, you can export them directly to pages to
share
your reports with colleagues.If you’re building advanced workflows utilizing
Galaxy’s powerful sub-workflows for reusable workflow components, then
you’ll
be pleased to know you can now automatically update those to the latest
version

Remote Files
--
The Remote Files interface is an absolutely fantastic new way to
browse your data. Inside Galaxy there is a new, abstract way to
reference files locally and on other servers. This lets us provide a
uniform interface to FTP servers, your Dropbox, public S3 buckets, and
more! You'll find this under Choose Remote Files in the upload
interface. This interface is also used for importing and exporting
histories.

Beta History Panel
-
The History Panel is getting a refresh and a huge performance boost in
the latest code. Does this sound exciting to you? Try it out now with
the history menu option "Use Beta History Panel". This is not its final
state but we'd love feedback from you, the users, on how you find it. It
features both performance and usability improvements. E.g. now you can
rename files without going into a separate menu, just double click the
dataset title!

Performance Improvements
-
This release includes many performance improvements for browsing and
searching Data libraries, improved job creation performance for certain
classes of tools and copying Histories, Datasets and Dataset Collections

Offload Zip Archive Creation to NGINX
---
Galaxy can now utilize the NGINX module mod_zip to assemble and compress
dataset collection and library dataset archives on the fly. This method is
more efficient and can create large archives without affecting the Galaxy
server process. The required setup is described in the admin documentation
for proxying Galaxy with NGINX.

Check out the full release notes
 for
a lot more details, there are many more
enhancements, new visualizations and bug fixes as well as instructions for
upgrading your Galaxy installation.

Thanks for using Galaxy!
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Release of Galaxy 20.09

[Marius van den Beek]

Dear Community,

The Galaxy Committers team is pleased
to announce the release of Galaxy 20.09.

The release announcement for developers and admins can be found at
https://docs.galaxyproject.org/en/master/releases/20.09_announce.html
and user facing release notes are at
https://docs.galaxyproject.org/en/master/releases/20.09_announce_user.html.
A few release highlights are:

GTN In Galaxy

The Galaxy Training Network tutorials can now be accessed from within Galaxy
using the graduation cap icon. For updated tutorials, tools will be
highlighted as blue buttons. When clicked, these buttons will hide the GTN
and
take you directly to the correct version of the correct tool within the
Galaxy
interface.

Plugin framework for uploading datasets
--
Galaxy administrators can now configure different sources
from which users may upload files. These include global or
user-specifc webdav servers, dropbox accounts as well as
FTP and regular filesystem locations. Developers can add
new types of sources by adding PyFileSystem2

compatible plugins.

Upload directly from the Tool Form
--
Datasets can now be uploaded directly from the Tool Form.

Workflow import from GA4GH TRS servers
-
Galaxy can now search and import workflows from GA4GH TRS servers,
such as Dockstore  and WorkflowHub
.
We hope that sharing workflows on these platforms will facilitate re-use
and collaboration.

Simplified workflow submission form

Galaxy now presents a simpler and cleaner interface for submitting workflows
that focuses on the parameters to set and datasets to choose.

Accelerated batch job creation and workflow step scheduling
-
Galaxy now batches database interactions for the creation of batch jobs.
For large batches of jobs this can speed up job creation by 100-fold or
more.

Check out the release notes for a lot more details - there are many
more enhancements, new visualizations and bug fixes as well as
instructions for upgrading your Galaxy installation.

Thanks for using Galaxy!
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Re: Expose local filesystem to Galaxy?

[Marius van den Beek]

Hi Sandra,

if the goal is to import data into galaxy from disk you can designate an
area as `user_library_import_dir` (see
https://docs.galaxyproject.org/en/master/admin/options.html#user-library-import-dir)
and
configure user_library_import_dir_auto_creation,
user_library_import_symlink_whitelist
and user_library_import_check_permissions as appropriate.
The import process can either copy datasets or use symlinks to these files
on disk.
 If the goal is to export files from Galaxy back to the cluster file
system you could use
https://toolshed.g2.bx.psu.edu/view/earlhaminst/export_to_cluster/9838eed606ad

I hope that helps,
Marius

On Fri, 2 Oct 2020 at 06:57, Sandra Maksimovic <
sandra.maksimo...@mcri.edu.au> wrote:

> Hi there,
>
> We are hosting an on-prem Galaxy instance in a HPC environment. Our
> research staff would like the ability to manipulate raw data that is
> located on the shared filesystem without having to manually upload it into
> Galaxy. Is this currently supported and would there be any related
> documentation?
>
> Thanks,
> Sandra
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Release of Galaxy 20.05

[Marius van den Beek]

Dear Community,

The Galaxy Committers team is pleased
to announce the release of Galaxy 20.05.

The release announcement for developers and admins can be found at
https://docs.galaxyproject.org/en/master/releases/20.05_announce.html
and user facing release notes are at
https://docs.galaxyproject.org/en/master/releases/20.05_announce_user.html.
A few release highlights are:

Many new Interactive Tools


Galaxy interactive tools allow a greater depth of analysis via access to
GUI-based tools inside an instance. 10 new interactive tools (some
previously
available only on the UseGalaxy.eu server) join the standard base toolset.
Improvements to the Higlass interactive tool now also allow for a more
in-depth
visualization of multiple datasets in various formats simultaneously.

Data Tables can now be backed by refgenie
--

Refgenie manages storage, access, and transfer of reference genome
resources.
Galaxy can now fill data tables that were received from a refgenie
installation.

Tool Shed is now Python3 ready
--

As the last component in the Galaxy Codebase the Tool Shed has been ported
to
Python 3. This concludes a 4 year effort to port the Galaxy codebase to
Python
3.

Workflow Editor and Workflow Run Form in Vue.js
--

The Workflow Editor and Workflow Run Form have been re-written in Vue.js.
While
the functionality has been preserved this lays the groundwork for creating
beautiful and reactive custom variants of these components in the future

Accelerated Galaxy Startup
-
Galaxy now caches expanded tool documents, delays creating the tool search
index until after startup and creates search indexes incrementally

Check out the release notes for a lot more details - there are many
more enhancements such as social login, tool recommendations,
visualizations, datatypes and bug fixes.

Thanks for using Galaxy!
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Re: Checkboxes issue (could not find an answer in the forum)

[Marius van den Beek]

Hi,

This was a bug on the dev branch a few weeks ago that was fixed by
https://github.com/galaxyproject/galaxy/pull/9667.
I would advise against deploying from the dev branch if this is supposed to
become a production server.
The last stable release is 20.01 at the moment.

Best,
Marius


On Fri, 22 May 2020 at 11:14,  wrote:

> Dear all,
> I am i the process to deploy a local instance of the galaxy server and all
> seems to work pretty ok.
> There is only one issue, so far, that is really puzzling me and that is
> "checkboxes". Can't seem to find a way to tick them!
> For example, I am trying to get the blast+ tool but there is no way I can
> get those options "visukaly" set. I can see that the flag is correctly
> raised when clicking the option, but the user cants see what was selected
> or not.
> Anyone had the same experience? Anyone has a solution to this?
>
> Here is a snippet of the code I could fish out.
> 
> 
> ::before
> blastp - Traditional BLASTP to compare a protein query to a protein
> database
> 
> 
> ::before
> blastp-fast - Use longer words for seeding, faster but less
> accurate
>  type="radio" name="field-uid-765" value="blastp-short">
> ::before
> blastp-short - BLASTP optimized for queries shorter than 30
> residues
> 
> 
> 
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Release of Galaxy 20.01

[Marius van den Beek]

Dear Community,

The Galaxy Committers team is pleased to announce the release of Galaxy
20.01.

The release announcement for developers and admins can be found at
https://docs.galaxyproject.org/en/master/releases/20.01_announce.html
and user facing release notes are at
https://docs.galaxyproject.org/en/master/releases/20.01_announce_user.html.

A few release highlights are:

Workflow Executions Menu


This menu can be reached from the User menu -> Workflow Invocations
and lists your recent Workflow executions, their status and links to
the Workflow Editor and the History with the results of the Workflow
run.

Enhanced workflow functionality
---

Workflows can now make use of optional datasets and optional
parameters, optional parameter default values and dataset inputs can
now be restricted to user-defined datatypes. The workflow editor and
execution engine have been enhanced to allow defining options to
choose from, and these options can be connected to compatible text,
integer, float, color and select tool parameters for more flexible and
universally reusable workflows

Galay Markdown Pages and Workflow Reports as PDF


An additional link has been added to Galaxy Markdown Pages and
Workflow reports that exports your document as a standalone PDF.

Screen Reader friendly Navigation


The top menu and the Workflow Editor have been improved for easier
navigation using Screen Readers and the Keyboard.

Email-notification for completed jobs
-

You can now select to be notified by email when a tool run has
finished.

Check out the release notes for a lot more details - there are many
more enhancements such as new interface elements, visualizations,
datatypes and bug fixes as well as instructions for upgrading your
Galaxy installation to Python 3.

Thanks for using Galaxy!
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Re: trouble running the new galaxy instance

[Marius van den Beek]

Hi Jochen,

can you post out the full logs ?

Jul 15 10:23:11 green-galaxy galaxy[5957]: yaml.scanner.ScannerError:
mappin...e
Jul 15 10:23:11 green-galaxy galaxy[5957]: in
"/opt/galaxy/galaxy/config/gal...0

probably means that you have syntax error in galaxy.yml, but the full logs
will tell you more.

Best,
Marius

On Tue, 23 Jul 2019 at 09:26, Jochen Bick  wrote:

> Hi
>
> since we updated some loc files our galaxy (19.01) does start but is
> unavailable in the browser.
>
> # systemctl status galaxy
> ? galaxy.service - SYSV: Galaxy http://galaxyproject.org/
>Loaded: loaded (/etc/rc.d/init.d/galaxy; bad; vendor preset: disabled)
>Active: active (exited) since Mon 2019-07-15 10:23:11 CEST; 1 weeks 0
> days ago
>  Docs: man:systemd-sysv-generator(8)
>   Process: 5840 ExecStop=/etc/rc.d/init.d/galaxy stop (code=exited,
> status=0/SUCCESS)
>   Process: 5957 ExecStart=/etc/rc.d/init.d/galaxy start (code=exited,
> status=0/SUCCESS)
>
> Jul 15 10:23:11 green-galaxy galaxy[5957]: File
> "/opt/galaxy/galaxy/.venv/li...n
> Jul 15 10:23:11 green-galaxy galaxy[5957]: self.fetch_more_tokens()
> Jul 15 10:23:11 green-galaxy galaxy[5957]: File
> "/opt/galaxy/galaxy/.venv/li...s
> Jul 15 10:23:11 green-galaxy galaxy[5957]: return self.fetch_value()
> Jul 15 10:23:11 green-galaxy galaxy[5957]: File
> "/opt/galaxy/galaxy/.venv/li...e
> Jul 15 10:23:11 green-galaxy galaxy[5957]: self.get_mark())
> Jul 15 10:23:11 green-galaxy galaxy[5957]: yaml.scanner.ScannerError:
> mappin...e
> Jul 15 10:23:11 green-galaxy galaxy[5957]: in
> "/opt/galaxy/galaxy/config/gal...0
> Jul 15 10:23:11 green-galaxy galaxy[5957]: ...done.
> Jul 15 10:23:11 green-galaxy systemd[1]: Started SYSV: Galaxy
> http://galaxyp
> Hint: Some lines were ellipsized, use -l to show in full.
>
>
> Has anyone a good hint?
>
> Cheers Jochen
>
> --
> ETH Zurich
> *Jochen Bick*
> Animal Physiology
> Institute of Agricultural Sciences
> Postal address: Universitätstrasse 2 / LFW B 58.1
> 8092 Zurich, Switzerland
> Office: Eschikon 27
> 8315 Lindau, Switzerland
>
> Phone +41 52 354 92 06
> jochen.b...@usys.ethz.ch 
> www.ap.ethz.ch
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Re: Error during tool installation: "inotify instance limit reached"

[Marius van den Beek]

Hi Christopher,

you might want to turn off `watch_tools` if you are using it,
this is only needed when using the pretty-much undocumented
feature where you dump tools into a directory without registering them
in any tool_conf.xml file. All other tool reloading happens by another
means.

Similarly, if you're using 19.05 and you don't manipulate the tool data
table
entries manually you can disable `watch_tool_data_dir`.

Finally this is usually a sign of allocating too many processes and threads
for your webservers/handlers/mules,
as each one starts their own watchdog. A common misconception is that this
makes Galaxy run faster.
but it actually just increases the amount of concurrent requests Galaxy can
handle.

Best,
Marius

On Wed, 10 Jul 2019 at 11:43, Previti <
christopher.prev...@dkfz-heidelberg.de> wrote:

> Dear all,
>
> I've been getting this error for a few days now, it basically prevents
> anything from being installed on the first go.
> Individual tools can be installed, but even then I have to try multiple
> times.
>
> Any ideas how to fix this?
>
> Cheers,
> Christopher
>
> The details from galaxy.log are (installing an older VCFtools package):
>
> 172.22.24.119 - - [10/Jul/2019:10:44:32 +0200] "POST
> /admin_toolshed/prepare_for_install HTTP/1.1" 200 -
> "
> http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install?change
>
> set_revisions=34a6b690e4b5_ids=2ba42187f2588c4b_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F
> 
> "
> "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0) Gecko/20100101
> Firefox/67.0"
> galaxy.tools.toolbox.base DEBUG 2019-07-10 10:44:55,592 Appending to
> tool panel section: NGS:VCFtools
> 172.22.24.119 - - [10/Jul/2019:10:44:58 +0200] "POST
> /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 -
> "http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare
> _for_install" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
> Gecko/20100101 Firefox/67.0"
> 172.22.24.119 - - [10/Jul/2019:10:45:01 +0200] "POST
> /admin_toolshed/repository_installation_status_updates HTTP/1.1" 200 -
> "http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare
> _for_install" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:67.0)
> Gecko/20100101 Firefox/67.0"
> 172.22.24.119 - - [10/Jul/2019:10:44:55 +0200] "POST
> /admin_toolshed/install_repositories HTTP/1.1" 500 -
> "
> http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/prepare_for_install
> "
> "Moz
> illa/5.0 (X11; Fedora; Linux x86_64; rv:67.0) Gecko/20100101 Firefox/67.0"
> *Error - : inotify instance limit reached*
> URL:
>
> http://dkfzgalaxy.inet.dkfz-heidelberg.de/admin_toolshed/install_repositories
> File '/opt/galaxy/galaxy/lib/galaxy/web/framework/middleware/error.py',
> line 154 in __call__
>   app_iter = self.application(environ, sr_checker)
> File
> '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/paste/recursive.py',
> line 85 in __call__
>   return self.application(environ, start_response)
> File
>
> '/opt/galaxy/galaxy/.venv/lib/python2.7/site-packages/paste/httpexceptions.py',
> line 640 in __call__
>   return self.application(environ, start_response)
> File '/opt/galaxy/galaxy/lib/galaxy/web/framework/base.py', line 143 in
> __call__
>   return self.handle_request(environ, start_response)
> File '/opt/galaxy/galaxy/lib/galaxy/web/framework/base.py', line 222 in
> handle_request
>   body = method(trans, **kwargs)
> File '/opt/galaxy/galaxy/lib/galaxy/web/framework/decorators.py', line
> 101 in decorator
>   return func(self, trans, *args, **kwargs)
> File
>
> '/opt/galaxy/galaxy/lib/galaxy/webapps/galaxy/controllers/admin_toolshed.py',
> line 575 in install_repositories
>   reinstalling=reinstalling,
> File
> '/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
> line 846 in install_repositories
>   tool_panel_section_mapping=tool_panel_section_mapping)
> File
> '/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
> line 893 in install_tool_shed_repository
>   tool_panel_section_mapping=tool_panel_section_mapping)
> File
> '/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/install_manager.py',
> line 544 in __handle_repository_contents
>   repository_tools_tups = irmm.get_repository_tools_tups()
> File
>
> '/opt/galaxy/galaxy/lib/tool_shed/galaxy_install/metadata/installed_repository_metadata_manager.py',
> line 75 in get_repository_tools_tups
>   tool = self.app.toolbox.load_tool(os.path.abspath(load_relative_path),
> guid=guid, use_cached=False)
> File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/base.py', line 766 in
> load_tool
>   self.watch_tool(tool)
> File '/opt/galaxy/galaxy/lib/galaxy/tools/toolbox/base.py', line 774 in
> watch_tool
>   self._tool_watcher.watch_file(tool.config_file, tool.id)
> File 

[galaxy-dev] Re: Workflow assigning columns on output causes "Uncaught exception in exposed API method"

[Marius van den Beek]

Hi Peter,

thanks for the report, I've opened a pull request to fix this at
https://github.com/galaxyproject/galaxy/pull/8023.
We'll backport this to 18.09 after acceptance.
As a (terrible) workaround you could download the workflow and change the
section that looks like
```
  "action_type": "ColumnSetAction",
  "output_name": "out_file1",
  "action_arguments": {
"chromCol": 1,
"endCol": 3,
"startCol": 2
  }
```
to
```
  "action_type": "ColumnSetAction",
  "output_name": "out_file1",
  "action_arguments": {
"chromCol": "1",
"endCol": "3",
"startCol": "2"
  }
```
(the "output_name" value may be different for each tool) and then upload it
back to the instance.

Hope that helps,
Marius

On Wed, 22 May 2019 at 10:36, Peter Briggs 
wrote:

> Dear Devs
>
> We've encountered a problem trying to run a workflow on our local Galaxy
> instance, which is resulting in an "uncaught exception in exposed API
> method" message displayed in the Galaxy window.
>
> The most trivial workflow exhibiting the problem consists of a single
> 'cut' task, where the output of the task is configured to change the
> datatype to "interval", and to explicitly assign the chrom, start and end
> columns to 1,2 and 3. Attempts to run this workflow generate the API
> exception.
>
> If the workflow is updated so that the column assignments are no longer
> attempted then the workflow is able to start without apparent problems.
>
> Our local Galaxy instance is a bit out of data (18.09) but I have been
> able to reproduce this error on Galaxy main so it looks like it also
> affects later versions. The failing workflow can be accessed on main via:
>
>
> https://usegalaxy.org/u/pjbriggs/w/cut-workflow-reassign-columns-fails-with-uncaught-api-exception
>
> The same workflow with the column assignment removed is accessible via:
>
> https://usegalaxy.org/u/pjbriggs/w/cut-workflow-no-column-reassignment
>
> (Main offers two variants of the 'cut' tool - as far as I can tell this
> problem happens with both.)
>
> Apologies if this has already been reported elsewhere - any suggestions
> for a workaround appreciated.
>
> Thanks,
>
> Peter
>
> --
> Peter Briggs peter.bri...@manchester.ac.uk
> Bioinformatics Core Facility University of Manchester
> B.1083 Michael Smith Bldg Tel: (0161) 2751482
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

[galaxy-dev] Re: tool installation problem

[Marius van den Beek]

Hi Matthias,

It might just have been deactivated, in which case you can either
activate or uninstall the repo in the admin panel -> Manage tools ->
Advanced Search -> Deactivated or Uninstalled.

Hope that helps,
Marius

On Tue, 26 Mar 2019 at 15:59, Matthias Bernt  wrote:

> Dear list,
>
> I have a problem during installing a tool version that I previously
> uninstalled. I get the message:
>
> `Revision 0696db066a5b of repository deseq2 owned by iuc has already
> been installed.`
>
> It seems to be uninstalled on the file system. Are there any entries
> that might be left in the DB?
>
> Thanks in advance.
>
> Cheers,
> Matthias
>
>
>
>
> --
>
> ---
> Matthias Bernt
> Bioinformatics Service
> Molekulare Systembiologie (MOLSYB)
> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
> Helmholtz Centre for Environmental Research GmbH - UFZ
> Permoserstraße 15, 04318 Leipzig, Germany
> Phone +49 341 235 482296,
> m.be...@ufz.de, www.ufz.de
>
> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ
> Helmholtz Centre for Environmental Research GmbH - UFZ
> Permoserstraße 15, 04318 Leipzig, Germany
> Phone +49 341 235 1269, Fax +49 341 235 1468 (optional)
> max.musterm...@ufz.de, www.ufz.de
>
> Sitz der Gesellschaft/Registered Office: Leipzig
> Registergericht/Registration Office: Amtsgericht Leipzig
> Handelsregister Nr./Trade Register Nr.: B 4703
> Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
> MinDirig Wilfried Kraus
> Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
> Prof. Dr. Georg Teutsch
> Administrative Geschäftsführerin/Administrative Managing Director:
> Dr. Sabine König
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   %(web_page_url)s
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  %(web_page_url)s

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Question using sftp to upload file to galaxy

[Marius van den Beek]

Hi Rui,

there's a fairly complete explanation and example in in
https://docs.galaxyproject.org/en/latest/admin/special_topics/ftp.html

Hope that helps,
Marius

On Mon, 28 Jan 2019 at 07:35, Rui Wang  wrote:

> Hey Folks,
>
> I tried a few times with different configurations, but none worked. Did
> anyone have the successful experience that could share? :-)
>
> Cheers,
> Rui
>
> On Sat, Jan 19, 2019 at 1:43 PM Rui Wang  wrote:
>
>> Hey Folks,
>>
>> I'm looking at the instructions of using ftp with proftpd. There is a
>> section talking about extending it to use sftp. However, the sample config
>> isn't comprehensive. I'm wondering if anyone has a working config for
>> reference?
>>
>> What's the setting of user and group? It says it should match the one in
>> the SQLNamedQuery, what does it mean exactly? I start proftpd as root, but
>> start galaxy as bioinfoadmin(normal user with sudo).
>>
>> Just fyi, my proftpd config module and config file are pasted below. I'm
>> working it out on a trial and error fashion, please feel free to point out
>> if anything is wrong!
>>
>> Cheers,
>> Rui
>>
>> modules:
>> $ sbin/proftpd -l
>> Compiled-in modules:
>>   mod_core.c
>>   mod_xfer.c
>>   mod_rlimit.c
>>   mod_auth_unix.c
>>   mod_auth.c
>>   mod_ls.c
>>   mod_log.c
>>   mod_site.c
>>   mod_delay.c
>>   mod_facts.c
>>   mod_sql.c
>>   mod_sql_postgres.c
>>   mod_sql_passwd.c
>>   mod_sftp.c
>>   mod_cap.c
>>
>> etc/proftpd.conf
>>
>> ServerTypestandalone
>>   # You must put this in a virtual host if you want it to listen on its
>> own port. VHost != Apache Vhost.
>>   
>> Port 
>> SFTPEngine on
>> AuthOrder mod_auth_unix.c mod_sql.c # If you don't do this you will
>> get weird disconnects
>> SFTPHostKey /etc/ssh/ssh_host_rsa_key
>> RequireValidShell no
>> MaxLoginAttempts 6
>> ServerName  "Galaxy SFTP"
>> DefaultServer   on
>> Umask   077
>> User bioinfoadmin
>> Group   bioinfoadmin
>> UseFtpUsers off
>> DefaultRoot ~
>> AllowOverwrite  on
>> AllowStoreRestart   on
>> SQLEngine   on
>> SQLGroupInfosftp_groups name id members
>>
>> # Do not authenticate against real (system) users
>> 
>> AuthPAM off
>> 
>>
>> # Common SQL authentication options
>> SQLPasswordEngine   on
>> SQLBackend  postgres
>> SQLConnectInfo  gal...@galaxy.my.org:5432 bioinfoadmin
>> dbpwd
>> SQLAuthenticate users
>>
>> # Configuration that handles PBKDF2 encryption
>> # Set up mod_sql to authenticate against the Galaxy database
>> SQLAuthTypesPBKDF2
>> SQLPasswordPBKDF2   SHA256 1 24
>> SQLPasswordEncoding base64
>> SQLPasswordUserSalt sql:/GetUserSalt
>>
>> # Define a custom query for lookup that returns a passwd-like entry.
>> Replace 512s with the UID and GID of the user running the Galaxy server
>> SQLUserInfo custom:/LookupGalaxyUser
>> SQLNamedQuery   LookupGalaxyUser SELECT "email, (CASE
>> WHEN substring(password from 1 for 6) = 'PBKDF2' THEN substring(password
>> from 38 for 69) ELSE password END) AS
>> password2,512,512,'/media/galaxy/galaxy/database/ftp/%U','/bin/bash' FROM
>> galaxy_user WHERE email='%U'"
>>
>> # Define custom query to fetch the password salt
>> SQLNamedQuery   GetUserSalt SELECT "(CASE WHEN SUBSTRING
>> (password from 1 for 6) = 'PBKDF2' THEN SUBSTRING (password from 21 for 16)
>> END) AS salt FROM galaxy_user WHERE email='%U'"
>>   
>>
>> # Don't use IPv6 support by default.
>> UseIPv6 off
>> MaxInstances30
>>
>> # To cause every FTP user to be "jailed" (chrooted) into their home
>> # directory, uncomment this line.
>> # Bar use of SITE CHMOD by default
>> 
>>   DenyAll
>> 
>>
>> # Bar use of RETR (download) since this is not a public file drop
>> 
>>   DenyAll
>> 
>> ~
>>
>> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] tools gone after switching to galaxy.yml

[Marius van den Beek]

Hi Matthias,

if your previous config had a separate database for tools you need to
use that. This is controlled by the install_database_connection setting.

Best,
Marius

On Wed, 19 Dec 2018 at 13:53, Matthias Bernt  wrote:

> Dear list,
>
> just tried to switch to galaxy.yml, which worked in general, but now
> tools that have been installed from the TS disappeared. In the logs I
> get for each tool:
>
> ```
> galaxy.tools.toolbox.base ERROR 2018-12-19 13:28:15,815
> [p:22877,w:0,m:0] [MainThread] Error reading tool from path:
>
> toolshed.g2.bx.psu.edu/repos/bgruening/text_processing/74a8bef53a00/text_processing/cat.xml
> Traceback (most recent call last):
>File "lib/galaxy/tools/toolbox/base.py", line 563, in _load_tool_tag_set
>  tool_shed_repository = self.get_tool_repository_from_xml_item(item,
> path)
>File "lib/galaxy/tools/toolbox/base.py", line 629, in
> get_tool_repository_from_xml_item
>  raise Exception(msg)
> Exception: Attempted to load tool shed tool, but the repository with
> name 'text_processing' from owner 'bgruening' was not found in database
> ```
>
> Any idea what could be wrong here? The connection to the DB seems to be
> functional, since I can see the users in the admin Panel.
>
> Cheers,
> Matthias
>
> --
>
> ---
> Matthias Bernt
> Bioinformatics Service
> Molekulare Systembiologie (MOLSYB)
> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
> Helmholtz Centre for Environmental Research GmbH - UFZ
> Permoserstraße 15, 04318 Leipzig, Germany
> Phone +49 341 235 482296,
> m.be...@ufz.de, www.ufz.de
>
> Sitz der Gesellschaft/Registered Office: Leipzig
> Registergericht/Registration Office: Amtsgericht Leipzig
> Handelsregister Nr./Trade Register Nr.: B 4703
> Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
> MinDirig Wilfried Kraus
> Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
> Prof. Dr. Dr. h.c. Georg Teutsch
> ---
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] number of elements in collections and multiple-true-params

[Marius van den Beek]

Hi Matthias,

I think you're right in that len($input_type.input_bam) is inconsistent,
we can implement __len__ for the collection case, but that is not done yet.

That said, unless your example is a toy example for a more complex scenario
the switch seems unnecessary, a multiple="true" data paramater can take
either multiple
datasets or a collection.

Best,
Marius
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Error after creating dataset list of bam files in v18.05

[Marius van den Beek]

Dear Christopher,

You need to copy the shipped datatypes_conf.xml.sample to the appropriate
location(s) for your installation.

Best,
Marius


On Tue, Oct 23, 2018, 11:08 AM Previti <
christopher.prev...@dkfz-heidelberg.de> wrote:

> Dear all,
>
> We're in the process of testing version 18.05 before updating our
> production server.
> The update process went fine but now I'm getting a "curious" error:
>
> Every time I make a dataset list of bam files the interfaces gives me
> following error when I try to run any tool:
>
> "Uncaught exception in exposed API method:"
>
> The error message in the galaxy.log is the following (here I just wanted
> to run hisat2, but it doesn't really matter which tool I try):
>
> 172.22.24.119 - - [19/Oct/2018:13:21:05 +0200] "POST /api/tools/
> toolshed.g2.bx.psu.edu/repos/iuc/hisat2/hisat2/2.0.5.2/build HTTP/1.1"
> 500 - "http://galaxy-test.inet.dkfz-heidelberg.de/?tool_id=toolshed.g2.bx.
> psu.edu%2Frepos%2Fiuc%2Fhisat2%2Fhisat2%2F2.0.5.2=2.0.5.2&__identifer=froqct2i1h"
> "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:62.0) Gecko/20100101
> Firefox/62.0"
> 172.22.24.119 - - [19/Oct/2018:13:21:07 +0200] "GET
> /api/histories/1cd8e2f6b131e891/contents?order=hid=dev=update_time-ge=2018-10-19T11%3A21%3A03.000Z
> HTTP/1.1" 200 -
> "http://galaxy-test.inet.dkfz-heidelberg.de/?tool_id=toolshed.g2.bx.psu.edu%2Frepos%2Fiuc%2Fhisat2%2Fhisat2%2F2.0.5.2=2.0.5.2&__identifer=froqct2i1h;
> 
> "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:62.0) Gecko/20100101
> Firefox/62.0"
> galaxy.datatypes.registry WARNING 2018-10-19 13:21:08,014 Datatype class
> not found for extension 'bai', which is used as target for conversion from
> datatype 'bam'
> galaxy.datatypes.registry WARNING 2018-10-19 13:21:08,015 Datatype class
> not found for extension 'bai', which is used as target for conversion from
> datatype 'bam'
> galaxy.datatypes.registry WARNING 2018-10-19 13:21:08,015 Datatype class
> not found for extension 'bai', which is used as target for conversion from
> datatype 'bam'
> galaxy.datatypes.registry WARNING 2018-10-19 13:21:08,015 Datatype class
> not found for extension 'bai', which is used as target for conversion from
> datatype 'bam'
> galaxy.datatypes.registry WARNING 2018-10-19 13:21:08,016 Datatype class
> not found for extension 'bai', which is used as target for conversion from
> datatype 'bam'
> galaxy.datatypes.registry WARNING 2018-10-19 13:21:08,016 Datatype class
> not found for extension 'bai', which is used as target for conversion from
> datatype 'bam'
> galaxy.datatypes.registry WARNING 2018-10-19 13:21:08,016 Datatype class
> not found for extension 'bai', which is used as target for conversion from
> datatype 'bam'
> galaxy.datatypes.registry WARNING 2018-10-19 13:21:08,019 Datatype class
> not found for extension 'bai', which is used as target for conversion from
> datatype 'bam'
> galaxy.web.framework.decorators ERROR 2018-10-19 13:21:08,038 Uncaught
> exception in exposed API method:
> Traceback (most recent call last):
>   File "/opt/galaxy/galaxy/lib/galaxy/web/framework/decorators.py", line
> 281, in decorator
> rval = func(self, trans, *args, **kwargs)
>   File "/opt/galaxy/galaxy/lib/galaxy/webapps/galaxy/api/tools.py", line
> 113, in build
> return tool.to_json(trans, kwd.get('inputs', kwd))
>   File "/opt/galaxy/galaxy/lib/galaxy/tools/__init__.py", line 1883, in
> to_json
> self.populate_model(request_context, self.inputs, state_inputs,
> tool_model['inputs'])
>   File "/opt/galaxy/galaxy/lib/galaxy/tools/__init__.py", line 1933, in
> populate_model
> tool_dict = input.to_dict(request_context)
>   File "/opt/galaxy/galaxy/lib/galaxy/tools/parameters/grouping.py", line
> 701, in to_dict
> cond_dict["cases"] = list(map(nested_to_dict, self.cases))
>   File "/opt/galaxy/galaxy/lib/galaxy/tools/parameters/grouping.py", line
> 699, in nested_to_dict
> return input.to_dict(trans)
>   File "/opt/galaxy/galaxy/lib/galaxy/tools/parameters/grouping.py", line
> 719, in to_dict
> when_dict["inputs"] = list(map(input_to_dict, self.inputs.values()))
>   File "/opt/galaxy/galaxy/lib/galaxy/tools/parameters/grouping.py", line
> 717, in input_to_dict
> return input.to_dict(trans)
>   File "/opt/galaxy/galaxy/lib/galaxy/tools/parameters/grouping.py", line
> 701, in to_dict
> cond_dict["cases"] = list(map(nested_to_dict, self.cases))
>   File "/opt/galaxy/galaxy/lib/galaxy/tools/parameters/grouping.py", line
> 699, in nested_to_dict
> return input.to_dict(trans)
>   File "/opt/galaxy/galaxy/lib/galaxy/tools/parameters/grouping.py", line
> 719, in to_dict
> when_dict["inputs"] = list(map(input_to_dict, self.inputs.values()))
>   File "/opt/galaxy/galaxy/lib/galaxy/tools/parameters/grouping.py", line
> 717, in input_to_dict
> return input.to_dict(trans)
>   File 

Re: [galaxy-dev] can't create user on a new instance

[Marius van den Beek]

Hi Rui,

The session token is used to prevent certain kinds of attacks. Your browser
may still have a session token for the old instance. You can delete the
cookies for this address in your browser and you should be issued a new and
valid token.

Hope that helps,
Marius

On Thu, Oct 11, 2018, 7:12 AM Qiang Gu  wrote:

> Hi Rui,
>
> Does the default database sqlite work for you? I guess there is certain
> mis-configuration in your
> postgresql settings or certain element in the config/galaxy.yml file is
> broken. Try to enter your postgresql database in terminal using the same
> login info and create a table to make sure you have proper privileges.
> Alternatively, using GalaxyKickStart (
> https://github.com/ARTbio/GalaxyKickStart) to deploy a server
> isn't that difficult and a postgresql database comes with it by default.
> Good luck!
>
> Thanks,
>
> -Qiang
>
> --
> *From:* galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on behalf
> of Rui Wang [ruiwang...@gmail.com]
> *Sent:* Wednesday, October 10, 2018 9:30 PM
> *To:* galaxy-dev@lists.galaxyproject.org
> *Subject:* Re: [galaxy-dev] can't create user on a new instance
>
> Hi Folks,
>
> I still could not create new user account with the same error message. Has
> anyone happened to see this before?
>
> Cheers,
> Rui
>
> On Tue, Oct 9, 2018 at 5:48 PM Rui Wang  wrote:
>
>> Hey Folks,
>>
>> I just installed a new 18.05 instance with a new postgresql db created.
>> However, the UI doesn't allow me to create any user. After I provided all
>> input information, it says
>>
>> galaxy.web.framework.webapp WARNING 2018-10-09 17:28:35,735
>> [p:32925,w:1,m:0] [uWSGIWorker1Core2] Wrong session token found, denying
>> request
>>
>> in the server log. Btw, I enabled 'allow to create user' and admin user
>> list is empty.
>>
>> I looked into it, and it's a function in
>> lib/galaxy/web/framework/webapp.py:
>>
>> def check_csrf_token(self):
>> session_csrf_token =
>> self.request.params.get("session_csrf_token", None)
>> problem = False
>> if not session_csrf_token:
>> log.warning("No session_csrf_token set, denying request.")
>> problem = True
>> elif session_csrf_token != self.session_csrf_token:
>> log.warning("Wrong session token found, denying request.")
>> problem = True
>>
>> if problem:
>> return self.show_warn_message("Failed to authorize action.")
>>
>> Could someone give me a hand? :-)
>>
>> Cheers,
>> Rui
>>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Issue with deseq2

[Marius van den Beek]

Hi Christopher,

due to the old version of Galaxy you need to manually update Conda (conda
update conda)
and explicitly set the current channel order in your galaxy.ini file, this
should be
conda_ensure_channels = iuc,conda-forge,bioconda,defaults

With these set you can remove the exisiting conda environment for deseq2
(conda env remove -n
mulled-v1-dbc1e05d0db06d983572e7eca0df68af225d8f398b6e11dd4d586075518c4101)
and then re-install deseq2.
Once you update to a newer Galaxy version you can comment the
conda_ensure_channels setting again
and continue using the default channel order that Galaxy ships.

Hope that helps,
Marius


On Tue, 18 Sep 2018 at 12:18, Christopher Previti <
christopher.prev...@dkfz-heidelberg.de> wrote:

> Dear all,
>
> I have been trying to get deseq2 (latest version 17,  2018-09-05 from iuc;
> we are still running on galaxy version 17.09) to work properly but to no
> avail. Older versions have other errors, I have focused on the latest
> version.
>
> I get this cryptic error when I run it:
>
> /opt/galaxy/galaxy/database/dependencies/_conda/envs/mulled-v1-dbc1e05d0db06d983572e7eca0df68af225d8f398b6e11dd4d586075518c4101/lib/R/bin/exec/R:
>  symbol lookup error:
> /opt/galaxy/galaxy/database/dependencies/_conda/envs/mulled-v1-dbc1e05d0db06d983572e7eca0df68af225d8f398b6e11dd4d586075518c4101/lib/R/bin/exec/../../lib/../../libreadline.so.6:
>  undefined symbol: PC
>
> /opt/galaxy/galaxy/database/dependencies/_conda/envs/mulled-v1-dbc1e05d0db06d983572e7eca0df68af225d8f398b6e11dd4d586075518c4101/lib/R/bin/exec/R:
>  symbol lookup error:
> /opt/galaxy/galaxy/database/dependencies/_conda/envs/mulled-v1-dbc1e05d0db06d983572e7eca0df68af225d8f398b6e11dd4d586075518c4101/lib/R/bin/exec/../../lib/../../libreadline.so.6:
>  undefined symbol: PC
>
> Does anybody know what to do about this?
>
> Thanks and best regards,
>
> Christopher
> --
> *Dr. Christopher Previti*
> Genomics and Proteomics Core Facility
> High Throughput Sequencing (W190)
> Bioinformatician
>
> German Cancer Research Center (DKFZ)
> Foundation under Public Law
> Im Neuenheimer Feld 580
> 69120 Heidelberg
> Germany
> Room: B2.102 (INF580/TP3)
> Phone: +49 6221 42-4661
>
> christopher.prev...@dkfz.de 
> www.dkfz.de
>
> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
> VAT-ID No.: DE143293537
>
> Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
> Personen bestimmt, an die sie adressiert ist.
> Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
> Informationen enthalten. Sollten Sie nicht
> der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender
> und löschen Sie die Mitteilung.
> Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
> untersagt.
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] java options

[Marius van den Beek]

Hi Matthias,

if a tool prints something to the stderr that is indeed considered to
indicate a failure, and it's the default mostly because
we don't want to break legacy tools that make this assumption. If you
include a profile version in your tool
or change your command section to  this will more reasonably determine
if a tool failed based on the exit code.
To me this sounds like something that really should be fixed in the tool as
presumably this affects every deployer.

Best,
Marius

On Mon, 27 Aug 2018 at 11:52, Matthias Bernt  wrote:

> Dear list,
>
> I'm struggling to set java options for galaxy tools. Currently I use
> ` in my job_conf.xml and in the script
> I explored two ways to set java options:
>
> `_JAVA_OPTIONS="-Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp"`
>
> But then java prints "Picked up _JAVA_OPTIONS: ..." to stderr which is
> be interpreted as error by some tools. If I'm correct this is currently
> the default .. I have seen this happening, but can't remember the name
> of the tool.
>
> So I switched to this 'trick':
>
> `alias java='java -Xmx5G -Xms1G -Djava.io.tmpdir=/work/songalax/tmp'`
>
> but the problem is that the tool script is called and not sourced and
> therefore aliases are not used (could this be changed?). Furthermore,
> for tools which explicitly set the java options my settings would be
> ignored anyway (an example is the MSGFPlusAdapter which calles `java
> -Xmx3500m` .. so my -Xmx is overwritten .. I passing GALAXY_MEMORY_MB to
> the corresponding parameter might be a solution for this tool).
>
> Any thought or suggestions on how to set the parameters in a production
> environment are very welcome.
>
> Cheers,
> Matthias
>
>
>
>
> --
>
> ---
> Matthias Bernt
> Bioinformatics Service
> Molekulare Systembiologie (MOLSYB)
> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
> Helmholtz Centre for Environmental Research GmbH - UFZ
> Permoserstraße 15, 04318 Leipzig, Germany
> Phone +49 341 235 482296,
> m.be...@ufz.de, www.ufz.de
>
> Sitz der Gesellschaft/Registered Office: Leipzig
> Registergericht/Registration Office: Amtsgericht Leipzig
> Handelsregister Nr./Trade Register Nr.: B 4703
> Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
> MinDirig Wilfried Kraus
> Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
> Prof. Dr. Dr. h.c. Georg Teutsch
> Administrative Geschäftsführerin/ Administrative Managing Director:
> Prof. Dr. Heike Graßmann
> ---
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Unable to install tools from toolshed/mercurial issue in release_18.05 instance

[Marius van den Beek]

Hi Peter,

Recent galaxy releases are using the `hg` command that should be
automatically installed along with other galaxy dependencies.
If you're running galaxy in a virtualenv then that virtualenv should have
the `hg` script in the bin folder.
Depending on how you start galaxy you may need to add the virtualenv's
`bin` folder to the `PATH`.

Hope that helps,
Marius

On 25 June 2018 at 07:51, Peter Briggs 
wrote:

> Dear devs
>
> I've encountered what appears to be a subtle bug with release 18.05, which
> breaks the installation of tools from the toolshed, and appears to be a
> result of not having mercurial (hg) available in /usr/bin on the system
> that Galaxy is installed on (in this case Scientific Linux 6.5).
>
> When attempting to install a tool (e.g. devteam/fastqc) from the main
> toolshed via the admin interface, after clicking "install" the tool
> installation status goes immediately to "Error". The tool repository isn't
> cloned to "shed_tools" and no dependencies are installed.
>
> I've been unable to find any error messages in the logs. However,
> attempting to install via the API does return the message:
>
> Error cloning repository: [Errno 2] No such file or directory
>
> which comes from the "clone_repository" function in
> lib/tool_shed/util/hg_util.py (when something goes wrong with the "hg clone
> ..." command).
>
> Installing Mercurial 1.3 via yum on the server and attempting tool
> installation again gives a slightly different error via the API:
>
> Error cloning repository: Command '['hg', 'clone', '-r', u'17', u'
> https://toolshed.g2.bx.psu.edu/repos/devteam/fastqc',
> u'/XX/shed_tools/toolshed.g2.bx.psu.edu/repos/
> devteam/fastqc/c15237684a01/fastqc']' returned non-zero exit status 255
> Output was:
> abort: No such file or directory: /XX/shed_tools/
> toolshed.g2.bx.psu.edu/repos/devteam/fastqc/c15237684a01/fastqc
>
> Uninstalling the system Mercurial and instead installing version 3.7.3 and
> making a link from /usr/bin/hg seems to fix the problem, and tools can be
> installed without problems.
>
> I couldn't find any evidence of this being reported before, and I don't
> know if I've missed some configuration detail which means that Galaxy isn't
> picking up hg from its virtualenv instead of /usr/bin.
>
> Has anyone else encountered this problem? and is there a fix/workaround
> for it (other than horrible links to /usr/bin/hg)?
>
> Any advice gratefully received!
>
> Best wishes
>
> Peter
>
> --
> Peter Briggs peter.bri...@manchester.ac.uk
> Bioinformatics Core Facility University of Manchester
> B.1083 Michael Smith Bldg Tel: (0161) 2751482
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] pbs/torque jobs

[Marius van den Beek]

Great, good to know this is all working!

On 14 June 2018 at 19:35, Briand, Sheldon (NRC/CNRC) <
sheldon.bri...@canada.ca> wrote:

> Hi Marius,
>
>
>
> I want to thank you for your help with this.  The problem was I that I was
> using the wrong deployment.  I have left the settings to the default
> deployment but I just switched to uWSGI + Mules and that has fixed my
> problem.
>
>
>
> Thanks!
>
> -Sheldon
>
>
>
> *From:* Marius van den Beek [mailto:m.vandenb...@gmail.com]
> *Sent:* Thursday, June 14, 2018 2:12 PM
>
> *To:* Briand, Sheldon (NRC/CNRC) 
> *Cc:* galaxy-dev@lists.galaxyproject.org
> *Subject:* Re: [galaxy-dev] pbs/torque jobs
>
>
>
> Hi Sheldon,
>
>
>
> It is normal to have a pbs.py file in your virtualenv. This is a file
> provided by pbs-python, which galaxy makes use of.
>
> This is not the same as the file you just modified, which contains the
> code galaxy uses to submit and check the status of PBS jobs,
>
> using this library.
>
>
>
> Assuming you restarted galaxy in between (forgot to mention that) this
> probably means
>
> that your job handlers are somehow not active.
>
> How are you starting galaxy and which setup are you trying to achieve ?
>
> An up to date and extensive documentation of the available options is here:
>
> https://docs.galaxyproject.org/en/latest/admin/scaling.
> html#deployment-options
>
>
>
> Marius
>
>
>
> On 14 June 2018 at 18:28, Briand, Sheldon (NRC/CNRC) <
> sheldon.bri...@canada.ca> wrote:
>
> I put that code in the file and reran the job.  None of those status
> messages show up.  So I went looking and I see a pbs.py in my .venv
> directory that is twice as large as the file I changed and it doesn’t look
> at all like the current pbs.py.
>
>
>
> Probably the wrong pbs.py is being called.  I’ll need to check the docs on
> venv setup again.
>
>
>
> Thanks,
>
> -Sheldon
>
>
>
> *From:* Marius van den Beek [mailto:m.vandenb...@gmail.com]
> *Sent:* Thursday, June 14, 2018 12:27 PM
>
>
> *To:* Briand, Sheldon (NRC/CNRC) 
> *Cc:* galaxy-dev@lists.galaxyproject.org
> *Subject:* Re: [galaxy-dev] pbs/torque jobs
>
>
>
> Could you try running a job again with the following commit ?
>
> https://github.com/mvdbeek/galaxy/commit/5bdd86104ae6b8923b4bed62ef3b5e
> e448c62f17
>
>
>
> This just logs some more things that might be helpful in diagnosing what's
> going on.
>
>
>
> On 14 June 2018 at 16:27, Briand, Sheldon (NRC/CNRC) <
> sheldon.bri...@canada.ca> wrote:
>
> Marius,
>
>
>
> Thanks for the help with this!
>
>
>
> Below is what I am seeing in the logs but it only gets as far as saying
> the job is queued (the status of the job doesn’t change, even though torque
> actually finishes the job successfully-I have something configured wrong):
>
> Dispatching to pbs runner
>
> galaxy.jobs DEBUG 2018-06-14 11:12:22,356 [p:63562,w:2,m:0]
> [JobHandlerQueue.monitor_thread] (16117) Persisting job destination
> (destination id: pbs_default)
>
> galaxy.tools.deps DEBUG 2018-06-14 11:12:22,775 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] Using dependency htseq version 0.9.1 of type conda
>
> galaxy.tools.deps DEBUG 2018-06-14 11:12:22,775 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] Using dependency samtools version 1.7 of type
> conda
>
> galaxy.jobs.command_factory INFO 2018-06-14 11:12:22,857 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] Built script [/BigData/galaxy/galaxy-dist/
> database/jobs_directory/016/16117/tool_script.sh] for tool command [[
> "$CONDA_DEFAULT_ENV" = "/BigData/galaxy/galaxy-dist/
> conda_deps/envs/mulled-v1-c9f488ec0e9a96bed61dcc2e074b26
> ce37ed596751861ff368fd824a2a5f11d4" ] ||
>
> galaxy.jobs.runners DEBUG 2018-06-14 11:12:23,143 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] (16117) command is: rm -rf working; mkdir -p
> working; cd working; /BigData/galaxy/galaxy-dist/
> database/jobs_directory/016/16117/tool_script.sh; return_code=$?; cd
> '/BigData/galaxy/galaxy-dist/database/jobs_directory/016/16117';
>
> galaxy.jobs.runners.pbs DEBUG 2018-06-14 11:12:23,208 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] (16117) submitting file
> /BigData/galaxy/galaxy-dist/database/pbs/16117.sh
>
> galaxy.jobs.runners.pbs DEBUG 2018-06-14 11:12:23,214 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] (16117) queued in default queue as 1434.locahost
>
> galaxy.jobs DEBUG 2018-06-14 11:12:23,214 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] (16117) Persisting job destination (destination
> id: pbs_default)
>
>
>
>
>
>
>
>
>
>
>
> *From:* Marius 

Re: [galaxy-dev] pbs/torque jobs

[Marius van den Beek]

Hi Sheldon,

It is normal to have a pbs.py file in your virtualenv. This is a file
provided by pbs-python, which galaxy makes use of.
This is not the same as the file you just modified, which contains the code
galaxy uses to submit and check the status of PBS jobs,
using this library.

Assuming you restarted galaxy in between (forgot to mention that) this
probably means
that your job handlers are somehow not active.
How are you starting galaxy and which setup are you trying to achieve ?
An up to date and extensive documentation of the available options is here:
https://docs.galaxyproject.org/en/latest/admin/scaling.html#deployment-options


Marius

On 14 June 2018 at 18:28, Briand, Sheldon (NRC/CNRC) <
sheldon.bri...@canada.ca> wrote:

> I put that code in the file and reran the job.  None of those status
> messages show up.  So I went looking and I see a pbs.py in my .venv
> directory that is twice as large as the file I changed and it doesn’t look
> at all like the current pbs.py.
>
>
>
Probably the wrong pbs.py is being called.  I’ll need to check the docs on
> venv setup again.
>

>
> Thanks,
>
> -Sheldon
>
>
>
> *From:* Marius van den Beek [mailto:m.vandenb...@gmail.com]
> *Sent:* Thursday, June 14, 2018 12:27 PM
>
> *To:* Briand, Sheldon (NRC/CNRC) 
> *Cc:* galaxy-dev@lists.galaxyproject.org
> *Subject:* Re: [galaxy-dev] pbs/torque jobs
>
>
>
> Could you try running a job again with the following commit ?
>
> https://github.com/mvdbeek/galaxy/commit/5bdd86104ae6b8923b4bed62ef3b5e
> e448c62f17
>
>
>
> This just logs some more things that might be helpful in diagnosing what's
> going on.
>
>
>
> On 14 June 2018 at 16:27, Briand, Sheldon (NRC/CNRC) <
> sheldon.bri...@canada.ca> wrote:
>
> Marius,
>
>
>
> Thanks for the help with this!
>
>
>
> Below is what I am seeing in the logs but it only gets as far as saying
> the job is queued (the status of the job doesn’t change, even though torque
> actually finishes the job successfully-I have something configured wrong):
>
> Dispatching to pbs runner
>
> galaxy.jobs DEBUG 2018-06-14 11:12:22,356 [p:63562,w:2,m:0]
> [JobHandlerQueue.monitor_thread] (16117) Persisting job destination
> (destination id: pbs_default)
>
> galaxy.tools.deps DEBUG 2018-06-14 11:12:22,775 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] Using dependency htseq version 0.9.1 of type conda
>
> galaxy.tools.deps DEBUG 2018-06-14 11:12:22,775 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] Using dependency samtools version 1.7 of type
> conda
>
> galaxy.jobs.command_factory INFO 2018-06-14 11:12:22,857 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] Built script [/BigData/galaxy/galaxy-dist/
> database/jobs_directory/016/16117/tool_script.sh] for tool command [[
> "$CONDA_DEFAULT_ENV" = "/BigData/galaxy/galaxy-dist/
> conda_deps/envs/mulled-v1-c9f488ec0e9a96bed61dcc2e074b26
> ce37ed596751861ff368fd824a2a5f11d4" ] ||
>
> galaxy.jobs.runners DEBUG 2018-06-14 11:12:23,143 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] (16117) command is: rm -rf working; mkdir -p
> working; cd working; /BigData/galaxy/galaxy-dist/
> database/jobs_directory/016/16117/tool_script.sh; return_code=$?; cd
> '/BigData/galaxy/galaxy-dist/database/jobs_directory/016/16117';
>
> galaxy.jobs.runners.pbs DEBUG 2018-06-14 11:12:23,208 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] (16117) submitting file
> /BigData/galaxy/galaxy-dist/database/pbs/16117.sh
>
> galaxy.jobs.runners.pbs DEBUG 2018-06-14 11:12:23,214 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] (16117) queued in default queue as 1434.locahost
>
> galaxy.jobs DEBUG 2018-06-14 11:12:23,214 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] (16117) Persisting job destination (destination
> id: pbs_default)
>
>
>
>
>
>
>
>
>
>
>
> *From:* Marius van den Beek [mailto:m.vandenb...@gmail.com]
> *Sent:* Thursday, June 14, 2018 6:29 AM
>
>
> *To:* Briand, Sheldon (NRC/CNRC) 
> *Cc:* galaxy-dev@lists.galaxyproject.org
> *Subject:* Re: [galaxy-dev] pbs/torque jobs
>
>
>
> So I was able to simulate a torque environment with https://github.com/
> aiidateam/torquessh_base-docker
>
> and that seemed to have worked fine.
>
>
>
> I did have to install the torque headers, activate galaxy's virtualenv and
> install https://github.com/ehiggs/pbs-python.
>
>
>
> You should be seeing something like:
>
>
>
> ```
>
> galaxy.jobs.runners.pbs DEBUG 2018-06-14 09:23:31,549 [p:1944,w:1,m:0]
> [PBSRunner.work_thread-2] (2) submitting file /home/app/galaxy/database/pbs/
> 2.sh
>
> galaxy.jobs.runners.pbs DEBUG 2018-06-14 09:23:31,551 [p:1944,w:1,m:0]
> [PBSRunner.work_th

Re: [galaxy-dev] pbs/torque jobs

[Marius van den Beek]

Could you try running a job again with the following commit ?
https://github.com/mvdbeek/galaxy/commit/5bdd86104ae6b8923b4bed62ef3b5ee448c62f17

This just logs some more things that might be helpful in diagnosing what's
going on.

On 14 June 2018 at 16:27, Briand, Sheldon (NRC/CNRC) <
sheldon.bri...@canada.ca> wrote:

> Marius,
>
>
>
> Thanks for the help with this!
>
>
>
> Below is what I am seeing in the logs but it only gets as far as saying
> the job is queued (the status of the job doesn’t change, even though torque
> actually finishes the job successfully-I have something configured wrong):
>
> Dispatching to pbs runner
>
> galaxy.jobs DEBUG 2018-06-14 11:12:22,356 [p:63562,w:2,m:0]
> [JobHandlerQueue.monitor_thread] (16117) Persisting job destination
> (destination id: pbs_default)
>
> galaxy.tools.deps DEBUG 2018-06-14 11:12:22,775 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] Using dependency htseq version 0.9.1 of type conda
>
> galaxy.tools.deps DEBUG 2018-06-14 11:12:22,775 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] Using dependency samtools version 1.7 of type
> conda
>
> galaxy.jobs.command_factory INFO 2018-06-14 11:12:22,857 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] Built script [/BigData/galaxy/galaxy-dist/
> database/jobs_directory/016/16117/tool_script.sh] for tool command [[
> "$CONDA_DEFAULT_ENV" = "/BigData/galaxy/galaxy-dist/
> conda_deps/envs/mulled-v1-c9f488ec0e9a96bed61dcc2e074b26
> ce37ed596751861ff368fd824a2a5f11d4" ] ||
>
> galaxy.jobs.runners DEBUG 2018-06-14 11:12:23,143 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] (16117) command is: rm -rf working; mkdir -p
> working; cd working; /BigData/galaxy/galaxy-dist/
> database/jobs_directory/016/16117/tool_script.sh; return_code=$?; cd
> '/BigData/galaxy/galaxy-dist/database/jobs_directory/016/16117';
>
> galaxy.jobs.runners.pbs DEBUG 2018-06-14 11:12:23,208 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] (16117) submitting file
> /BigData/galaxy/galaxy-dist/database/pbs/16117.sh
>
> galaxy.jobs.runners.pbs DEBUG 2018-06-14 11:12:23,214 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] (16117) queued in default queue as 1434.locahost
>
> galaxy.jobs DEBUG 2018-06-14 11:12:23,214 [p:63562,w:2,m:0]
> [PBSRunner.work_thread-1] (16117) Persisting job destination (destination
> id: pbs_default)
>
>
>
>
>
>
>
>
>
>
>
> *From:* Marius van den Beek [mailto:m.vandenb...@gmail.com]
> *Sent:* Thursday, June 14, 2018 6:29 AM
>
> *To:* Briand, Sheldon (NRC/CNRC) 
> *Cc:* galaxy-dev@lists.galaxyproject.org
> *Subject:* Re: [galaxy-dev] pbs/torque jobs
>
>
>
> So I was able to simulate a torque environment with https://github.com/
> aiidateam/torquessh_base-docker
>
> and that seemed to have worked fine.
>
>
>
> I did have to install the torque headers, activate galaxy's virtualenv and
> install https://github.com/ehiggs/pbs-python.
>
>
>
> You should be seeing something like:
>
>
>
> ```
>
> galaxy.jobs.runners.pbs DEBUG 2018-06-14 09:23:31,549 [p:1944,w:1,m:0]
> [PBSRunner.work_thread-2] (2) submitting file /home/app/galaxy/database/pbs/
> 2.sh
>
> galaxy.jobs.runners.pbs DEBUG 2018-06-14 09:23:31,551 [p:1944,w:1,m:0]
> [PBSRunner.work_thread-2] (2) queued in default queue as 3.605046c8289c
>
> galaxy.jobs DEBUG 2018-06-14 09:23:31,552 [p:1944,w:1,m:0]
> [PBSRunner.work_thread-2] (2) Persisting job destination (destination id:
> local)
>
> galaxy.jobs.runners.pbs DEBUG 2018-06-14 09:23:31,855 [p:1944,w:1,m:0]
> [Dummy-5] (2/3.605046c8289c) PBS job state changed from N to R
>
> galaxy.jobs.runners.pbs DEBUG 2018-06-14 09:23:33,994 [p:1944,w:1,m:0]
> [Dummy-5] (2/3.605046c8289c) PBS job has left queue
>
> galaxy.model.metadata DEBUG 2018-06-14 09:23:34,110 [p:1944,w:1,m:0]
> [PBSRunner.work_thread-3] loading metadata from file for:
> HistoryDatasetAssociation 2
>
> galaxy.jobs INFO 2018-06-14 09:23:34,218 [p:1944,w:1,m:0]
> [PBSRunner.work_thread-3] Collecting metrics for Job 2
>
> galaxy.jobs DEBUG 2018-06-14 09:23:34,234 [p:1944,w:1,m:0]
> [PBSRunner.work_thread-3] job 2 ended (finish() executed in (189.200 ms))
>
> ```
>
>
>
> in your logs.
>
>
>
> Let me know how this goes.
>
>
>
> Best,
>
> Marius
>
>
>
> On 14 June 2018 at 10:15, Marius van den Beek 
> wrote:
>
> Hi Sheldon,
>
>
>
> I'm not sure what the issue could be, the PBS runner hasn't been updated
> in ~3 years,
>
> but of course many things around it have been.
>
> Could you set galaxy's logging level to debug if it isn't already and
> check the logs ?
>
> (https://github.com/galaxyproject/galaxy/blob/release_18.05/confi

Re: [galaxy-dev] pbs/torque jobs

[Marius van den Beek]

So I was able to simulate a torque environment with
https://github.com/aiidateam/torquessh_base-docker
and that seemed to have worked fine.

I did have to install the torque headers, activate galaxy's virtualenv and
install https://github.com/ehiggs/pbs-python.

You should be seeing something like:

```
galaxy.jobs.runners.pbs DEBUG 2018-06-14 09:23:31,549 [p:1944,w:1,m:0]
[PBSRunner.work_thread-2] (2) submitting file
/home/app/galaxy/database/pbs/2.sh
galaxy.jobs.runners.pbs DEBUG 2018-06-14 09:23:31,551 [p:1944,w:1,m:0]
[PBSRunner.work_thread-2] (2) queued in default queue as 3.605046c8289c
galaxy.jobs DEBUG 2018-06-14 09:23:31,552 [p:1944,w:1,m:0]
[PBSRunner.work_thread-2] (2) Persisting job destination (destination id:
local)
galaxy.jobs.runners.pbs DEBUG 2018-06-14 09:23:31,855 [p:1944,w:1,m:0]
[Dummy-5] (2/3.605046c8289c) PBS job state changed from N to R
galaxy.jobs.runners.pbs DEBUG 2018-06-14 09:23:33,994 [p:1944,w:1,m:0]
[Dummy-5] (2/3.605046c8289c) PBS job has left queue
galaxy.model.metadata DEBUG 2018-06-14 09:23:34,110 [p:1944,w:1,m:0]
[PBSRunner.work_thread-3] loading metadata from file for:
HistoryDatasetAssociation 2
galaxy.jobs INFO 2018-06-14 09:23:34,218 [p:1944,w:1,m:0]
[PBSRunner.work_thread-3] Collecting metrics for Job 2
galaxy.jobs DEBUG 2018-06-14 09:23:34,234 [p:1944,w:1,m:0]
[PBSRunner.work_thread-3] job 2 ended (finish() executed in (189.200 ms))
```

in your logs.

Let me know how this goes.

Best,
Marius

On 14 June 2018 at 10:15, Marius van den Beek 
wrote:

> Hi Sheldon,
>
> I'm not sure what the issue could be, the PBS runner hasn't been updated
> in ~3 years,
> but of course many things around it have been.
> Could you set galaxy's logging level to debug if it isn't already and
> check the logs ?
> (https://github.com/galaxyproject/galaxy/blob/release_18.05/config/galaxy.
> yml.sample#L895)
> When you are submitting a job in galaxy what messages are you seeing in
> the logs ?
>
> Best,
> Marius
>
> On 13 June 2018 at 22:07, Briand, Sheldon (NRC/CNRC) <
> sheldon.bri...@canada.ca> wrote:
>
>> Hi Marius,
>>
>>
>>
>> The PBS runner and the user is the galaxy user.  I do not use the run as
>> real user option.  I haven’t been using drmaa_external_runjob_script.
>> This setup worked for my old 17.05 and previous versions of galaxy.
>>
>>
>>
>> Thanks,
>>
>> -Sheldon
>>
>>
>>
>> *From:* Marius van den Beek [mailto:m.vandenb...@gmail.com]
>> *Sent:* Wednesday, June 13, 2018 4:58 PM
>> *To:* Briand, Sheldon (NRC/CNRC) 
>> *Cc:* galaxy-dev@lists.galaxyproject.org
>> *Subject:* Re: [galaxy-dev] pbs/torque jobs
>>
>>
>>
>> Hi Sheldon,
>>
>>
>>
>> is there anything particular about your job configuration, e.g.
>>
>> are you you using the run as real user option or are you using
>>
>> the drmaa_external_runjob_script option ?
>>
>> Are you using the drmaa or the PBS runner ?
>>
>>
>>
>> Best,
>>
>> Marius
>>
>>
>>
>> On 13 June 2018 at 21:13, Briand, Sheldon (NRC/CNRC) <
>> sheldon.bri...@canada.ca> wrote:
>>
>> Hi,
>>
>>
>>
>> I have upgraded to Galaxy 18.05 (from 17.05).  I am running a torque job
>> scheduler (version 6.02).
>>
>>
>>
>> When I submit a job through galaxy and I look in the admin/manage jobs
>> section I see that the job has been submitted successfully.  It shows that
>> the job is queued and waiting to run.  On the cluster I see that the job
>> runs and goes to completion.  However, the status in galaxy continue to
>> show that the job is waiting to run and the status never gets updated.  I’m
>> using postgres as my database and I am running through a nginx proxy.
>>
>>
>>
>> I see no errors in my galaxy.log file.
>>
>>
>>
>> I switched from galaxy.ini to galaxy.yml and from paste to uwsgi.  Is
>> this a configuration problem on my end?  Where should I be looking?
>>
>>
>>
>> Thanks,
>>
>> -Sheldon
>>
>>
>>
>> Sheldon Briand
>>
>> Computer Systems and Applications Analyst
>>
>> National Research Council/Government of Canada
>>
>> sheldon.bri...@canada.ca/ Tel: (902) 426-1677
>>
>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/
>>
>>
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] pbs/torque jobs

[Marius van den Beek]

Hi Sheldon,

I'm not sure what the issue could be, the PBS runner hasn't been updated in
~3 years,
but of course many things around it have been.
Could you set galaxy's logging level to debug if it isn't already and check
the logs ?
(
https://github.com/galaxyproject/galaxy/blob/release_18.05/config/galaxy.yml.sample#L895
)
When you are submitting a job in galaxy what messages are you seeing in the
logs ?

Best,
Marius

On 13 June 2018 at 22:07, Briand, Sheldon (NRC/CNRC) <
sheldon.bri...@canada.ca> wrote:

> Hi Marius,
>
>
>
> The PBS runner and the user is the galaxy user.  I do not use the run as
> real user option.  I haven’t been using drmaa_external_runjob_script.
> This setup worked for my old 17.05 and previous versions of galaxy.
>
>
>
> Thanks,
>
> -Sheldon
>
>
>
> *From:* Marius van den Beek [mailto:m.vandenb...@gmail.com]
> *Sent:* Wednesday, June 13, 2018 4:58 PM
> *To:* Briand, Sheldon (NRC/CNRC) 
> *Cc:* galaxy-dev@lists.galaxyproject.org
> *Subject:* Re: [galaxy-dev] pbs/torque jobs
>
>
>
> Hi Sheldon,
>
>
>
> is there anything particular about your job configuration, e.g.
>
> are you you using the run as real user option or are you using
>
> the drmaa_external_runjob_script option ?
>
> Are you using the drmaa or the PBS runner ?
>
>
>
> Best,
>
> Marius
>
>
>
> On 13 June 2018 at 21:13, Briand, Sheldon (NRC/CNRC) <
> sheldon.bri...@canada.ca> wrote:
>
> Hi,
>
>
>
> I have upgraded to Galaxy 18.05 (from 17.05).  I am running a torque job
> scheduler (version 6.02).
>
>
>
> When I submit a job through galaxy and I look in the admin/manage jobs
> section I see that the job has been submitted successfully.  It shows that
> the job is queued and waiting to run.  On the cluster I see that the job
> runs and goes to completion.  However, the status in galaxy continue to
> show that the job is waiting to run and the status never gets updated.  I’m
> using postgres as my database and I am running through a nginx proxy.
>
>
>
> I see no errors in my galaxy.log file.
>
>
>
> I switched from galaxy.ini to galaxy.yml and from paste to uwsgi.  Is this
> a configuration problem on my end?  Where should I be looking?
>
>
>
> Thanks,
>
> -Sheldon
>
>
>
> Sheldon Briand
>
> Computer Systems and Applications Analyst
>
> National Research Council/Government of Canada
>
> sheldon.bri...@canada.ca/ Tel: (902) 426-1677
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] pbs/torque jobs

[Marius van den Beek]

Hi Sheldon,

is there anything particular about your job configuration, e.g.
are you you using the run as real user option or are you using
the drmaa_external_runjob_script option ?
Are you using the drmaa or the PBS runner ?

Best,
Marius

On 13 June 2018 at 21:13, Briand, Sheldon (NRC/CNRC) <
sheldon.bri...@canada.ca> wrote:

> Hi,
>
>
>
> I have upgraded to Galaxy 18.05 (from 17.05).  I am running a torque job
> scheduler (version 6.02).
>
>
>
> When I submit a job through galaxy and I look in the admin/manage jobs
> section I see that the job has been submitted successfully.  It shows that
> the job is queued and waiting to run.  On the cluster I see that the job
> runs and goes to completion.  However, the status in galaxy continue to
> show that the job is waiting to run and the status never gets updated.  I’m
> using postgres as my database and I am running through a nginx proxy.
>
>
>
> I see no errors in my galaxy.log file.
>
>
>
> I switched from galaxy.ini to galaxy.yml and from paste to uwsgi.  Is this
> a configuration problem on my end?  Where should I be looking?
>
>
>
> Thanks,
>
> -Sheldon
>
>
>
> Sheldon Briand
>
> Computer Systems and Applications Analyst
>
> National Research Council/Government of Canada
>
> sheldon.bri...@canada.ca/ Tel: (902) 426-1677
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Samtools running on submit-node and as a slurm job

[Marius van den Beek]

Hi Christopher,

If your galaxy is configured to set metadata externally then you shouldn't
see samtools index being run on the node that runs galaxy
(On a side-note galaxy will use pysam starting from release 18.01).
One possibility is that the metadata setting failed, and it has been tried
again internally.
You can control this with the `retry_metadata_internally` option in your
galaxy.yml / galaxy.ini
file.

Best,
Marius

On 24 April 2018 at 15:55, Previti 
wrote:

> Dear all,
>
> We found something strange with our Galaxy instance...
>
> Our Galaxy server is configured as a submit node using slurm to submit to
> our cluster.
>
> This works fine for the most part, but today we saw a samtools index job
> running "locally" on the submit node instead of a  cluster node.
>
> Is there any way of stopping this behavior? The submit nodes are not
> configured for data-intensive computation and everything runs significantly
> slower.
>
> I was running bamtools (splitting a bam-file into mapped into unmapped) at
> the time and the output bam-files were being indexed.
>
> Thanks and best regards,
>
> Christopher
> --
> *Dr. Christopher Previti*
> Genomics and Proteomics Core Facility
> High Throughput Sequencing (W190)
> Bioinformatician
>
> German Cancer Research Center (DKFZ)
> Foundation under Public Law
> Im Neuenheimer Feld 580
> 69120 Heidelberg
> Germany
> Room: B2.102 (INF580/TP3)
> Phone: +49 6221 42-4661
>
> christopher.prev...@dkfz.de 
> www.dkfz.de
>
> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
> VAT-ID No.: DE143293537
>
> Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
> Personen bestimmt, an die sie adressiert ist.
> Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
> Informationen enthalten. Sollten Sie nicht
> der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender
> und löschen Sie die Mitteilung.
> Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
> untersagt.
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Problems installing Galaxy 18.01

[Marius van den Beek]

Hi Peter,

does the tool show up in the tool panel (Looks like you've installed it
into "Get Data")?
What commit are you on ? There were some last minute bugfixes to the
release_18.01
prior to the release, so if the checkout is a few days old an upgrade (and
an uninstall/install cycle) might help.

Also you mentioned the lack of the upload_store in nginx in nginx, you need
to install nginx-extras, not nginx.

Hope that helps,
Marius

On 18 March 2018 at 21:17, Peter van Heusden  wrote:

> Hi there
>
> My newly installed Galaxy 18.01 server seems to be unable to install
> bowtie (the devteam version). I'm documenting my installation progress at:
>
> http://pvh.wp.sanbi.ac.za/2018/03/18/galaxy-18-01-install/
>
> As noted in the last paragraph:
>
> "My next test was to try and install the bowtie2 tool from the toolshed.
> This failed, with the tool status going from New to Error with not much
> interesting left in the logs."
>
> The log during the time when I'm trying to install bowtie2 is:
>
>
>1. [pid: 8856|app: 0|req: 35/48] 10.8.0.2 () {48 vars in 1108 bytes} [Sun
>Mar 18 19:14:55 2018] GET /admin_toolshed/manage_repository?id=
>f597429621d6eb2b=install => generated 650 bytes in 26 msecs (
>HTTP/1.1 302) 3headers in 269 bytes (1 switches on core 1)
>2. 10.8.0.2 - - [18/Mar/2018:19:14:57 +] "GET
>/admin_toolshed/manage_repository?id=f597429621d6eb2b=install
>HTTP/1.1" 302 - "https://galaxy.sanbi.ac.za/admin_toolshed/manage_
>repository?id=f597429621d6eb2b" "Mozilla/5.0 (X11; Linux x86_64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/
>537.36"
>3. [pid: 8856|app: 0|req: 37/49] 10.8.0.2 () {48 vars in 1108 bytes} [Sun
>Mar 18 19:14:57 2018] GET /admin_toolshed/manage_repository?id=
>f597429621d6eb2b=install => generated 650 bytes in 24 msecs (
>HTTP/1.1 302) 3headers in 269 bytes (1 switches on core 0)
>4. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,375 [p:8856,w:2,m:0] [
>uWSGIWorker2Core2] Dependency bowtie2 not found.
>5. galaxy.tools.deps DEBUG 2018-03-18 19:14:59,376 [p:8856,w:2,m:0] [
>uWSGIWorker2Core2] Dependency samtools not found.
>6. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [
>uWSGIWorker2Core3] Dependency bowtie2 not found.
>7. galaxy.tools.deps DEBUG 2018-03-18 19:15:00,028 [p:8856,w:2,m:0] [
>uWSGIWorker2Core3] Dependency samtools not found.
>8. 10.8.0.2 - - [18/Mar/2018:19:14:55 +] "GET
>/admin_toolshed/prepare_for_install?changeset_revisions=
>dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F%
>2Ftoolshed.g2.bx.psu.edu %2F HTTP/1.1
>" 200 - "https://galaxy.sanbi.ac.za/admin_toolshed/manage_
>repository?id=f597429621d6eb2b" "Mozilla/5.0 (X11; Linux x86_64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/
>537.36"
>9. [pid: 8856|app: 0|req: 38/50] 10.8.0.2 () {48 vars in 1274 bytes} [Sun
>Mar 18 19:14:55 2018] GET /admin_toolshed/prepare_for_
>install?changeset_revisions=dc1639b66f12_ids=
>126c0918b5459666_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F
>=> generated 19062 bytes in 7001 msecs (HTTP/1.1 200) 2 headers in 73
> bytes (1 switches on core 2)
>10. 10.8.0.2 - - [18/Mar/2018:19:14:57 +] "GET
>/admin_toolshed/prepare_for_install?changeset_revisions=
>dc1639b66f12_ids=126c0918b5459666_shed_url=https%3A%2F%
>2Ftoolshed.g2.bx.psu.edu %2F HTTP/1.1
>" 200 - "https://galaxy.sanbi.ac.za/admin_toolshed/manage_
>repository?id=f597429621d6eb2b" "Mozilla/5.0 (X11; Linux x86_64)
> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/
>537.36"
>11. [pid: 8856|app: 0|req: 38/51] 10.8.0.2 () {48 vars in 1274 bytes} [Sun
>Mar 18 19:14:57 2018] GET /admin_toolshed/prepare_for_
>install?changeset_revisions=dc1639b66f12_ids=
>126c0918b5459666_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F
>=> generated 19062 bytes in 5987 msecs (HTTP/1.1 200) 2 headers in 73
> bytes (1 switches on core 3)
>12. galaxy.tools.toolbox.base DEBUG 2018-03-18 19:15:14,128 [p:8846,w:1
>,m:0] [uWSGIWorker1Core1] Appending to tool panel section: Get Data
>13. galaxy.tools.toolbox.base DEBUG 2018-03-18 19:15:14,143 [p:8846,w:1
>,m:0] [uWSGIWorker1Core1] Appending to tool panel section: Get Data
>14. 10.8.0.2 - - [18/Mar/2018:19:15:11 +] "POST
>/admin_toolshed/prepare_for_install HTTP/1.1" 200 - "
>https://galaxy.sanbi.ac.za/admin_toolshed/prepare_for_
>install?changeset_revisions=dc1639b66f12_ids=
>126c0918b5459666_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F
>
> "
>"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
> 

Re: [galaxy-dev] Ensuring Python 2.7 when resolving conda dependencies for tool

[Marius van den Beek]

It's coming from the defaults channel, I didn't check if this is actually
working,
it may very well not be compatible with the remaining packages form
conda-forge.
AFAIK you can't specify packages to install via the regular conda commands
(though you can use pip in a conda environemnt to do this manually).

Hope that helps,
Marius

On 5 March 2018 at 17:51, Peter Cock <p.j.a.c...@googlemail.com> wrote:

> I stand corrected. Looking closer, there are conda packages
> for both Python 2 and 3, e.g.
>
> https://anaconda.org/conda-forge/biopython/files
>
> However, in this case you're asking for an older Biopython
> which has to date not been packaged in conda-forge or
> bioconda, so I presume in Marius example it comes from
> PyPI via pip?
>
>
> Peter
>
> On Mon, Mar 5, 2018 at 4:46 PM, Marius van den Beek
> <m.vandenb...@gmail.com> wrote:
> > This should actually work properly if you install the dependencies via
> > the Manage dependencies page in the admin menu or if you install the tool
> > via the tool shed.
> > This translates more or less to the following conda command:
> > ```
> > $ conda create -n mulled-v1 python=2.7 biopython=1.65 -c iuc -c
> > bioconda -c conda-forge -c defaults
> > Fetching package metadata ...
> > Solving package specifications: .
> >
> > Package plan for installation in environment
> > /Users/mvandenb/miniconda3/envs/test_resolution:
> >
> > The following NEW packages will be INSTALLED:
> >
> > biopython:   1.65-np19py27_0
> > ca-certificates: 2018.1.18-0  conda-forge
> > certifi: 2018.1.18-py27_0 conda-forge
> > intel-openmp:2018.0.0-h8158457_8
> > libgfortran: 3.0.1-h93005f0_2
> > mkl: 2018.0.1-hfbd8650_4
> > ncurses: 5.9-10   conda-forge
> > numpy:   1.9.3-py27hb3dd696_3
> > openssl: 1.0.2n-0 conda-forge
> > pip: 9.0.1-py27_1 conda-forge
> > python:  2.7.14-4 conda-forge
> > readline:7.0-0conda-forge
> > setuptools:  38.5.1-py27_0conda-forge
> > sqlite:  3.20.1-2 conda-forge
> > tk:  8.6.7-0  conda-forge
> > wheel:   0.30.0-py27_2conda-forge
> > zlib:1.2.11-0 conda-forge
> > ```
> >
> > So given a recent conda (latest version should work) and the correct
> channel
> > order
> > it should work.
> >
> > Best,
> > Marius
> >
> > On 5 March 2018 at 17:39, Peter Cock <p.j.a.c...@googlemail.com> wrote:
> >>
> >> Tricky, as I understand it the conda python packages are specific to
> >> the conda version of Python - in this case Python 3 not 2.
> >>
> >> It might actually be simpler to fix pal_finder/pal_filter.py to cope
> >> with Python 3 - is the code online somewhere, I could probably cast an
> >> eye over it.
> >>
> >> Peter
> >>
> >> On Mon, Mar 5, 2018 at 4:28 PM, Peter Briggs
> >> <peter.bri...@manchester.ac.uk> wrote:
> >> > Hello
> >> >
> >> > I'm in the process of updating our local Galaxy tools to use conda
> >> > dependency resolution, and I've hit a snag with a tool that requires
> Python
> >> > 2.7 along with the Python 2.7-compatible version of Biopython 1.65.
> >> >
> >> > I'd assumed that if I explicitly used the following in the
> >> > "requirements" section of the tool XML:
> >> >
> >> > python
> >> > biopython requirement>
> >> >
> >> > that the biopython install would respect the specified Python version,
> >> > and that the command execution environment would be based on Python
> 2.7.
> >> >
> >> > But in practice it appears I'm getting Python3 as I'm seeing errors:
> >> >
> >> >   File "/galaxy-tools/tools/pal_finder/pal_filter.py",
> line
> >> > 129
> >> > print "\n~~"
> >> >^
> >> > SyntaxError: Missing parentheses in call to 'print'
> >> >
> >> > This is happening both when running the tool tests via the planemo
> >> > tests, and also when I install the tool from a local toolshed
> instance.
> >> >
> >> > Can anyone advise how to coerce the Python dependencies to be

Re: [galaxy-dev] Ensuring Python 2.7 when resolving conda dependencies for tool

[Marius van den Beek]

This should actually work properly if you install the dependencies via
the Manage dependencies page in the admin menu or if you install the tool
via the tool shed.
This translates more or less to the following conda command:
```
$ conda create -n mulled-v1 python=2.7 biopython=1.65 -c iuc -c
bioconda -c conda-forge -c defaults
Fetching package metadata ...
Solving package specifications: .

Package plan for installation in environment
/Users/mvandenb/miniconda3/envs/test_resolution:

The following NEW packages will be INSTALLED:

biopython:   1.65-np19py27_0
ca-certificates: 2018.1.18-0  conda-forge
certifi: 2018.1.18-py27_0 conda-forge
intel-openmp:2018.0.0-h8158457_8
libgfortran: 3.0.1-h93005f0_2
mkl: 2018.0.1-hfbd8650_4
ncurses: 5.9-10   conda-forge
numpy:   1.9.3-py27hb3dd696_3
openssl: 1.0.2n-0 conda-forge
pip: 9.0.1-py27_1 conda-forge
python:  2.7.14-4 conda-forge
readline:7.0-0conda-forge
setuptools:  38.5.1-py27_0conda-forge
sqlite:  3.20.1-2 conda-forge
tk:  8.6.7-0  conda-forge
wheel:   0.30.0-py27_2conda-forge
zlib:1.2.11-0 conda-forge
```

So given a recent conda (latest version should work) and the correct
channel order
it should work.

Best,
Marius

On 5 March 2018 at 17:39, Peter Cock  wrote:

> Tricky, as I understand it the conda python packages are specific to
> the conda version of Python - in this case Python 3 not 2.
>
> It might actually be simpler to fix pal_finder/pal_filter.py to cope
> with Python 3 - is the code online somewhere, I could probably cast an
> eye over it.
>
> Peter
>
> On Mon, Mar 5, 2018 at 4:28 PM, Peter Briggs
>  wrote:
> > Hello
> >
> > I'm in the process of updating our local Galaxy tools to use conda
> dependency resolution, and I've hit a snag with a tool that requires Python
> 2.7 along with the Python 2.7-compatible version of Biopython 1.65.
> >
> > I'd assumed that if I explicitly used the following in the
> "requirements" section of the tool XML:
> >
> > python
> > biopython
> >
> > that the biopython install would respect the specified Python version,
> and that the command execution environment would be based on Python 2.7.
> >
> > But in practice it appears I'm getting Python3 as I'm seeing errors:
> >
> >   File "/galaxy-tools/tools/pal_finder/pal_filter.py", line
> 129
> > print "\n~~"
> >^
> > SyntaxError: Missing parentheses in call to 'print'
> >
> > This is happening both when running the tool tests via the planemo
> tests, and also when I install the tool from a local toolshed instance.
> >
> > Can anyone advise how to coerce the Python dependencies to be
> 2.7-compatible?
> >
> > Thanks in advance for any help,
> >
> > Best wishes
> >
> > Peter
> >
> > --
> > Peter Briggs peter.bri...@manchester.ac.uk
> > Bioinformatics Core Facility University of Manchester
> > B.1083 Michael Smith Bldg Tel: (0161) 2751482
> >
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >   https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> >   http://galaxyproject.org/search/
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Text file busy and conda

[Marius van den Beek]

Hi John,

when you say tool autoinstalled via conda do you mean that the dependencies
were automatically installed during the repository installation process,
or that you have `conda_auto_install = True` in your galaxy.ini ?

The latter is not working reliably while the dependencies have not finished
the installation process.
To avoid this you can make sure that the dependency is correctly installed
before running the tool
(the dependency in the admin panel -> Manage tool dependencies is green and
no conda process is still active).

Best,
Marius

On 30 January 2018 at 03:15, John Letaw  wrote:

> Hi all,
>
>
>
> I’m receiving the following error trying to access tools autoinstalled via
> conda:
>
>
>
> Fatal error: Exit code 126 ()
>
> ~/galaxy/database/jobs_directory/000/38/tool_script.sh: line 41:
> ~/galaxy/database/dependencies/_conda/envs/mulled-v1-
> bb83f93efd111e042823e53ddfa74c32d81ba74cceca9445dfddfc9e940ff738/bin/samtools:
> Text file busy
>
>
>
> So, something is happening too fast I guess, meaning a process is
> attempting access while this is being created.  Any ideas on how I can
> diagnose this?  Not seeing any other errors floating around the logs.
> Maybe we are dealing with a lustre file locking issue?
>
>
>
> Thanks,
>
> John
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Error at the end of a BWA job

[Marius van den Beek]

Hi Frederic,

samtools couldn't be found, which is required for creating the bam index
files
during the metadata setting.
You can install it globally or in the admin panel -> Manage tool
dependencies -> Find the "Set External Metadata" tool
and hit "Install checked dependencies via Conda".

Best,
Marius

On 17 January 2018 at 15:23, SAPET, Frederic 
wrote:

> Hello
>
>
>
> I’ve just installed BWA ((Galaxy Version 0.7.17.3) , Galaxy itself is On
> branch release_17.09).
>
>
>
> The first run is not a success.
>
>
>
> I can see that the BAM is created, but job is tagged as fail (Unable to
> finish job).
>
>
>
> It looks like close to this issue: https://github.com/
> galaxyproject/galaxy/issues/2083
>
> I’ve checked, samtools is in the path.
>
>
>
> Do you have any ideas ?
>
>
>
> Thank you
>
>
>
> Fred
>
>
>
> Here is what I see in the log:
>
>
>
> galaxy.jobs.runners.drmaa DEBUG 2018-01-17 10:57:23,787
> (7062/86331.universe) state change: job finished normally
>
> galaxy.model.metadata DEBUG 2018-01-17 10:57:23,993 setting metadata
> externally failed for HistoryDatasetAssociation 11537: 'invalidated'
>
> galaxy.jobs.runners ERROR 2018-01-17 10:57:24,108 (7062/86331.universe)
> Job wrapper finish method failed
>
> Traceback (most recent call last):
>
>   File "/softs/bioinfo/galaxy-prod/lib/galaxy/jobs/runners/__init__.py",
> line 630, in finish_job
>
> job_state.job_wrapper.finish(stdout, stderr, exit_code)
>
>   File "/softs/bioinfo/galaxy-prod/lib/galaxy/jobs/__init__.py", line
> 1287, in finish
>
> dataset.datatype.set_meta(dataset, overwrite=False)
>
>   File "/softs/bioinfo/galaxy-prod/lib/galaxy/datatypes/binary.py", line
> 424, in set_meta
>
> exit_code = subprocess.call(args=command, stderr=open(stderr_name,
> 'wb'))
>
>   File "/softs/add-ons/Python-2.7.12/lib/python2.7/subprocess.py", line
> 523, in call
>
> return Popen(*popenargs, **kwargs).wait()
>
>   File "/softs/add-ons/Python-2.7.12/lib/python2.7/subprocess.py", line
> 711, in __init__
>
> errread, errwrite)
>
>   File "/softs/add-ons/Python-2.7.12/lib/python2.7/subprocess.py", line
> 1343, in _execute_child
>
> raise child_exception
>
> OSError: [Errno 2] No such file or directory
>
>
>
> ---
>
> Frederic Sapet
>
> Bioinformatics Research Engineer
>
> BIOGEMMA - Upstream Genomics Group
>
> Centre de Recherche de Chappes
>
> CS 90126
>
> 63720 CHAPPES
>
> FRANCE
>
> Tel : +33 (0)4 73 67 88 54 <+33%204%2073%2067%2088%2054>
>
> Fax : +33 (0)4 73 67 88 99 <+33%204%2073%2067%2088%2099>
>
> E-mail : frederic.sa...@biogemma.com
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] tool publishing

[Marius van den Beek]

Hi Matthias,

Glad it works.
I guess a few characters are blacklisted, but generally you can choose freely.
You can see your public name (which is what should be used in the `name` field)
by logging in to the toolshed, User -> Preferences -> Manage your Information.

Marius

On 18 December 2017 at 11:20, Matthias Bernt  wrote:
> Hi Marius,
>
>> Also there is a discrepancy between the `name` you're using in the
>> .shed.yml and the name as displayed in the
>> toolshed, that is probably another issue to look at.
>
>
> That was the problem. Thanks a lot. Can the name of the owner be freely
> chosen? I remember that I specified a username when registering, but I could
> not remember (and could not find a way to find this information on the
> toolshed page).
>
> Best,
> Matthias
>
>
>> On 18 December 2017 at 10:49, Matthias Bernt  wrote:
>>>
>>> Dear peter,
>>>
>>> thanks for your comments. I tried to update:
>>>
>>> `planemo shed_update --shed_target testtoolshed`
>>>
>>> But planemo also complains about non existent repository:
>>>
>>> ```
>>> Repository [fasta_regex_finder] does not exist in the targeted Tool Shed.
>>> Failed to update repository 'fasta_regex_finder' as it does not exist on
>>> the
>>> test Tool Shed
>>> ```
>>>
>>> Cheers,
>>> Matthias
>>>
>>>
>>>
>>> On 18.12.2017 10:45, Peter Briggs wrote:


 Hello Matthias

 My reading of the planemo docs it was that "shed_create" just made the
 repository, and that it had to be followed by the "shed_update" command
 in
 order to upload the actual content. (The documentation could probably be
 clearer on this I think.)

 I'm not sure about the error from "shed_diff", maybe related to the
 repository not having any content.

 Best wishes

 Peter

 --
 Peter Briggs peter.bri...@manchester.ac.uk
 Bioinformatics Core Facility University of Manchester
 B.1083 Michael Smith Bldg Tel: (0161) 2751482


 
 From: galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on behalf
 of
 Matthias Bernt [m.be...@ufz.de]
 Sent: Monday, December 18, 2017 9:37 AM
 To: Galaxy Dev List
 Subject: [galaxy-dev] tool publishing

 Dear dev-list,

 I'm just trying to publish my first tool to the testtoolshed via
 planemo. I got some problem during the process. Maybe someone can
 kickstart me (I guess I just forgot something).

 I followed http://planemo.readthedocs.io/en/latest/publishing.html

 The step

 `planemo shed_create --shed_target testtoolshed`

 ```
 Repository created
 cd '/home/berntm/projects/mb-galaxy-tools/fasta_regex_finder' && git
 rev-parse HEAD
 cd '/home/berntm/projects/mb-galaxy-tools/fasta_regex_finder' && git
 diff --quiet
 Repository [fasta_regex_finder] does not exist in the targeted Tool
 Shed.
 ```

 When I check online the repository is there but no changeset seems
 available. Retrying fails (expected):

 `planemo shed_create --shed_target testtoolshed`

 ```
 Unexpected HTTP status code: 400: {"err_msg": "You already own a
 repository named fasta_regex_finder<\/b>, please choose a different
 name.", "err_code": 48}
 

 Also shed_diff fails

 `fasta_regex_finder git:(master) ✗ planemo shed_diff --shed_target
 testtoolshed`

 ```
 shed_diff: Repository [fasta_regex_finder] does not exist in the
 targeted Tool Shed.
 ```

 Any idea?

 Cheers,
 Matthias

 --

 ---
 Matthias Bernt
 Bioinformatics Service
 Molekulare Systembiologie (MOLSYB)
 Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
 Helmholtz Centre for Environmental Research GmbH - UFZ
 Permoserstraße 15, 04318 Leipzig, Germany
 Phone +49 341 235 482296,
 m.be...@ufz.de, www.ufz.de

 Sitz der Gesellschaft/Registered Office: Leipzig
 Registergericht/Registration Office: Amtsgericht Leipzig
 Handelsregister Nr./Trade Register Nr.: B 4703
 Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
 MinDirig Wilfried Kraus
 Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
 Prof. Dr. Dr. h.c. Georg Teutsch
 Administrative Geschäftsführerin/ Administrative Managing Director:
 Prof. Dr. Heike Graßmann
 ---
 ___
 Please keep all replies on the list by using "reply all"
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/

>>>
>>> --
>>>
>>> 

Re: [galaxy-dev] tool publishing

[Marius van den Beek]

Hi Matthias,

this really looks like it should have worked.
The repository has been created, but the revision is not installable.

Often times this is because of missing tool data table / loc file references,
but I see that your tool is not making use of any of these so they are
not necessary.

My typical command line invocation contains the API key, like so:

```
planemo shed_update -t testtoolshed --shed_key $MY_KEY
```

Perhaps something is going wrong when passing the credentials around in planemo,
so that only the repo creation phase succeeds. Maybe you'd have more
luck explicitly passing in the repo key ?

Also there is a discrepancy between the `name` you're using in the
.shed.yml and the name as displayed in the
toolshed, that is probably another issue to look at.

Marius

On 18 December 2017 at 10:49, Matthias Bernt  wrote:
> Dear peter,
>
> thanks for your comments. I tried to update:
>
> `planemo shed_update --shed_target testtoolshed`
>
> But planemo also complains about non existent repository:
>
> ```
> Repository [fasta_regex_finder] does not exist in the targeted Tool Shed.
> Failed to update repository 'fasta_regex_finder' as it does not exist on the
> test Tool Shed
> ```
>
> Cheers,
> Matthias
>
>
>
> On 18.12.2017 10:45, Peter Briggs wrote:
>>
>> Hello Matthias
>>
>> My reading of the planemo docs it was that "shed_create" just made the
>> repository, and that it had to be followed by the "shed_update" command in
>> order to upload the actual content. (The documentation could probably be
>> clearer on this I think.)
>>
>> I'm not sure about the error from "shed_diff", maybe related to the
>> repository not having any content.
>>
>> Best wishes
>>
>> Peter
>>
>> --
>> Peter Briggs peter.bri...@manchester.ac.uk
>> Bioinformatics Core Facility University of Manchester
>> B.1083 Michael Smith Bldg Tel: (0161) 2751482
>>
>>
>> 
>> From: galaxy-dev [galaxy-dev-boun...@lists.galaxyproject.org] on behalf of
>> Matthias Bernt [m.be...@ufz.de]
>> Sent: Monday, December 18, 2017 9:37 AM
>> To: Galaxy Dev List
>> Subject: [galaxy-dev] tool publishing
>>
>> Dear dev-list,
>>
>> I'm just trying to publish my first tool to the testtoolshed via
>> planemo. I got some problem during the process. Maybe someone can
>> kickstart me (I guess I just forgot something).
>>
>> I followed http://planemo.readthedocs.io/en/latest/publishing.html
>>
>> The step
>>
>> `planemo shed_create --shed_target testtoolshed`
>>
>> ```
>> Repository created
>> cd '/home/berntm/projects/mb-galaxy-tools/fasta_regex_finder' && git
>> rev-parse HEAD
>> cd '/home/berntm/projects/mb-galaxy-tools/fasta_regex_finder' && git
>> diff --quiet
>> Repository [fasta_regex_finder] does not exist in the targeted Tool Shed.
>> ```
>>
>> When I check online the repository is there but no changeset seems
>> available. Retrying fails (expected):
>>
>> `planemo shed_create --shed_target testtoolshed`
>>
>> ```
>> Unexpected HTTP status code: 400: {"err_msg": "You already own a
>> repository named fasta_regex_finder<\/b>, please choose a different
>> name.", "err_code": 48}
>> 
>>
>> Also shed_diff fails
>>
>> `fasta_regex_finder git:(master) ✗ planemo shed_diff --shed_target
>> testtoolshed`
>>
>> ```
>> shed_diff: Repository [fasta_regex_finder] does not exist in the
>> targeted Tool Shed.
>> ```
>>
>> Any idea?
>>
>> Cheers,
>> Matthias
>>
>> --
>>
>> ---
>> Matthias Bernt
>> Bioinformatics Service
>> Molekulare Systembiologie (MOLSYB)
>> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
>> Helmholtz Centre for Environmental Research GmbH - UFZ
>> Permoserstraße 15, 04318 Leipzig, Germany
>> Phone +49 341 235 482296,
>> m.be...@ufz.de, www.ufz.de
>>
>> Sitz der Gesellschaft/Registered Office: Leipzig
>> Registergericht/Registration Office: Amtsgericht Leipzig
>> Handelsregister Nr./Trade Register Nr.: B 4703
>> Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board:
>> MinDirig Wilfried Kraus
>> Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
>> Prof. Dr. Dr. h.c. Georg Teutsch
>> Administrative Geschäftsführerin/ Administrative Managing Director:
>> Prof. Dr. Heike Graßmann
>> ---
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>http://galaxyproject.org/search/
>>
>
> --
>
> ---
> Matthias Bernt
> Bioinformatics Service
> Molekulare Systembiologie (MOLSYB)
> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
> Helmholtz Centre for Environmental Research GmbH - UFZ
> Permoserstraße 15, 04318 Leipzig, Germany
> Phone +49 341 235 482296,
> 

Re: [galaxy-dev] Fwd: Job Script Integrity with GalaxyKickStart (galaxy-dev Digest, Vol 137, Issue 5)

[Marius van den Beek]

Hi John and Christophe,

What the job script integrity script does is checking that the script is
ready to be executed,
by setting the environment variable `ABC_TEST_JOB_SCRIPT_INTEGRITY_XYZ` to 1
and then executing the tool_script.sh script that contains the following
check:

```
if [ -n "$ABC_TEST_JOB_SCRIPT_INTEGRITY_XYZ" ]; then
exit 42
fi
```

So if the script is ready to execute it returns with the exit code 42.
Now this can take a few seconds over NFS (I guess that'd be true for lustre
as well).
This check is being run 35 times with a sleep of .25 seconds.

Unfortunately there was a bug in galaxy that would skip the sleep,
so the job integrity check would fail frequently. We fixed this in
https://github.com/galaxyproject/galaxy/pull/4720 and this has been
backported
up to galaxy release 16.07, so if you just get to the latest galaxy commit
on your branch
it *may* work again.

Now this has been broken for a long time, and it has never worked for me
on our current cluster. Should an update to galaxy not be enough,
you can disable this check with `check_job_script_integrity = False` in the
galaxy.ini or by adding
`-e GALAXY_CONFIG_CHECK_JOB_SCRIPT_INTEGRITY=False` if you're running
kickstart in docker.
I have not seen any drawback of disabling the integrity check on our
cluster.

Good luck,
Marius

On 7 November 2017 at 18:25, Christophe Antoniewski 
wrote:

> Hi John,
>
> Can you also raise an issue in https://github.com/ARTbio/G
> alaxyKickStart/issues ?
>
> In order to help, I will need to know the configuration of your
> GalaxyKickStart (the variables you modified in the playbook, group_vars and
> inventory_files).
>
> Did you use the cloud_setup role ? In that case Enis Afgan
> https://github.com/afgane may help.
>
> Best regards
>
> Chris
>
> Christophe Antoniewski
>
> Institut de Biologie Paris Seine 
> 9, Quai St Bernard, Boîte courrier 24
> 75252 Paris Cedex 05
> ARTbio  Bâtiment B, 7e étage, porte 725
>
> Tel +33 1 44 2
> *7 70 05*Mobile +33 6 68 60 51 50 <06%2068%2060%2051%2050>
>
> Pour accéder à la Plateforme
> Bâtiment B, 7e étage, Porte 725
> 
>
>
> https://twitter.com/ARTbio_IBPS
>
>
> 2017-11-07 18:00 GMT+01:00 :
>
>> Send galaxy-dev mailing list submissions to
>> galaxy-dev@lists.galaxyproject.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://lists.galaxyproject.org/listinfo/galaxy-dev
>> or, via email, send a message with subject or body 'help' to
>> galaxy-dev-requ...@lists.galaxyproject.org
>>
>> You can reach the person managing the list at
>> galaxy-dev-ow...@lists.galaxyproject.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of galaxy-dev digest..."
>>
>>
>> HEY!  This is important!  If you reply to a thread in a digest, please
>> 1. Change the subject of your response from "Galaxy-dev Digest Vol ..."
>> to the original subject for the thread.
>> 2. Strip out everything else in the digest that is not part of the thread
>> you are responding to.
>>
>> Why?
>> 1. This will keep the subject meaningful.  People will have some idea
>> from the subject line if they should read it or not.
>> 2. Not doing this greatly increases the number of emails that match
>> search queries, but that aren't actually informative.
>>
>> Today's Topics:
>>
>>1. Job Script Integrity (John Letaw)
>>
>>
>> --
>>
>> Message: 1
>> Date: Tue, 7 Nov 2017 03:20:49 +
>> From: "John Letaw" 
>> To: "galaxy-dev@lists.galaxyproject.org"
>> 
>> Subject: [galaxy-dev] Job Script Integrity
>> Message-ID: 
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hi all,
>>
>> I’m installing via GalaxyKickStart…
>>
>> I’m getting the following error:
>>
>> galaxy.jobs.runners ERROR 2017-11-06 19:14:05,263 (19) Failure preparing
>> job
>> Traceback (most recent call last):
>>   File 
>> "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/__init__.py",
>> line 175, in prepare_job
>> modify_command_for_container=modify_command_for_container
>>   File 
>> "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/runners/__init__.py",
>> line 209, in build_command_line
>> container=container
>>   File 
>> "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/command_factory.py",
>> line 84, in build_command
>> externalized_commands = __externalize_commands(job_wrapper,
>> external_command_shell, commands_builder, remote_command_params)
>>   File 
>> "/home/exacloud/lustre1/galaxydev/galaxyuser/lib/galaxy/jobs/command_factory.py",
>> line 143, in __externalize_commands
>> write_script(local_container_script, 

Re: [galaxy-dev] Metadata setting python dependencies

[Marius van den Beek]

Are you perhaps using a supervisor setup where you are setting the PATH ?
In that case you can remove /phengs/hpc_storage/home/
galaxy_hpc/galaxy-dist/.venv/bin from the PATH.
If you're referencing python you can directly use the path to
/phengs/hpc_storage/home/galaxy_hpc/galaxy-dist/.venv/bin/python.

Best,
Marius

On 25 October 2017 at 18:24, Ulf Schaefer <ulf.schae...@phe.gov.uk> wrote:

> Hi Marius
>
> This is very useful, but not quite there.
>
> I use in my job_conf.xml
>
> 
> /this/is/a/test
> 
>
> and I end up with this
>
> PATH=/phengs/hpc_storage/home/galaxy_hpc/galaxy-dist/.venv/b
> in:/this/is/a/test
>
> Any idea where that came from?
>
> Thanks
> Ulf
>
> On 25/10/17 16:05, Marius van den Beek wrote:
>
>> Hi Ulf,
>>
>> you can set a virtualenv to be used in your job_conf.xml file.
>> The following line would go into the `` tag:
>> ``
>>
>> Best,
>> Marius
>>
>>
>> On 25 October 2017 at 16:56, Ulf Schaefer <ulf.schae...@phe.gov.uk
>> <mailto:ulf.schae...@phe.gov.uk>> wrote:
>>
>> Hi again
>>
>> Update, sorry for spamming:
>>
>> It is not ALL jobs that fail. Only the ones that get dispatched to
>> our Grid Engine cluster (so nearly all).
>>
>> The reason appears to be that the path to the Python binary is added
>> to the PATH variable when the job arrives on the worker node:
>>
>> [galaxy_hpc@hpc1node25 galaxy-dist]$ export
>> PATH=/home/galaxy_hpc/galaxy-dist/.venv/bin:/tmp/5000354.1.r
>> esearchmid.q:/usr/local/bin:/bin:/usr/bin
>> [galaxy_hpc@hpc1node25 galaxy-dist]$ python
>> Could not find platform dependent libraries 
>> Consider setting $PYTHONHOME to [:]
>> Python 2.7.6 (default, Oct 25 2017, 12:28:23)
>> [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import json
>> Traceback (most recent call last):
>>   File "", line 1, in 
>> ImportError: No module named json
>>
>> This is the wrong python to be running there. It should just go with
>> the one in /usr/bin. If I change PATH, it works fine:
>>
>> [galaxy_hpc@hpc1node25 galaxy-dist]$ export
>> PATH=/tmp/5000354.1.researchmid.q:/usr/local/bin:/bin:/usr/
>> bin[galaxy_hpc@hpc1node25
>> galaxy-dist]$ python
>> Python 2.6.6 (r266:84292, Aug  9 2016, 06:11:56)
>> [GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import json
>> >>>
>>
>> Does anyone have an idea how I can stop this from happening? Where
>> is the PATH set before submitting the job?
>>
>> Thanks
>> Ulf
>>
>>
>>
>>
>>
>>
>> On 25/10/17 15:23, Ulf Schaefer wrote:
>>
>> Hi all
>>
>> After running the recommended update I am now on branch 17.09.
>> As part
>> of the update I set up a new virtual env running an additional
>> python
>> 2.7.6. My previous virtual env was running 2.6.6 which is a system
>> requirement for Centos 6.
>>
>> All dependencies were installed and Galaxy starts fine, but no
>> tools
>> run. The error is always a variation of this:
>>
>> Could not find platform dependent libraries 
>> Consider setting $PYTHONHOME to [:]
>> Traceback (most recent call last):
>>   File
>> "galaxy-dist/database/jobs_directory/000/418/418014/set_meta
>> data_4LcsZv.py",
>> line 1, in 
>> from galaxy_ext.metadata.set_metadata import set_metadata;
>> set_metadata()
>>   File "galaxy-dist/lib/galaxy_ext/metadata/set_metadata.py",
>> line 19,
>> in 
>> import json
>> ImportError: No module named json
>>
>> I put some debug output at the top of set_metadata.py and it is
>> indeed
>> running the correct python version from the virtual env.
>> Needless to say
>> I can import json just fine using that python version. Any ideas?
>>
>> Any he

Re: [galaxy-dev] Metadata setting python dependencies

[Marius van den Beek]

Hi Ulf,

you can set a virtualenv to be used in your job_conf.xml file.
The following line would go into the `` tag:
``

Best,
Marius


On 25 October 2017 at 16:56, Ulf Schaefer  wrote:

> Hi again
>
> Update, sorry for spamming:
>
> It is not ALL jobs that fail. Only the ones that get dispatched to our
> Grid Engine cluster (so nearly all).
>
> The reason appears to be that the path to the Python binary is added to
> the PATH variable when the job arrives on the worker node:
>
> [galaxy_hpc@hpc1node25 galaxy-dist]$ export PATH=/home/galaxy_hpc/galaxy-d
> ist/.venv/bin:/tmp/5000354.1.researchmid.q:/usr/local/bin:/bin:/usr/bin
> [galaxy_hpc@hpc1node25 galaxy-dist]$ python
> Could not find platform dependent libraries 
> Consider setting $PYTHONHOME to [:]
> Python 2.7.6 (default, Oct 25 2017, 12:28:23)
> [GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import json
> Traceback (most recent call last):
>   File "", line 1, in 
> ImportError: No module named json
>
> This is the wrong python to be running there. It should just go with the
> one in /usr/bin. If I change PATH, it works fine:
>
> [galaxy_hpc@hpc1node25 galaxy-dist]$ export PATH=/tmp/5000354.1.researchmi
> d.q:/usr/local/bin:/bin:/usr/bin[galaxy_hpc@hpc1node25 galaxy-dist]$
> python
> Python 2.6.6 (r266:84292, Aug  9 2016, 06:11:56)
> [GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import json
> >>>
>
> Does anyone have an idea how I can stop this from happening? Where is the
> PATH set before submitting the job?
>
> Thanks
> Ulf
>
>
>
>
>
>
> On 25/10/17 15:23, Ulf Schaefer wrote:
>
>> Hi all
>>
>> After running the recommended update I am now on branch 17.09. As part
>> of the update I set up a new virtual env running an additional python
>> 2.7.6. My previous virtual env was running 2.6.6 which is a system
>> requirement for Centos 6.
>>
>> All dependencies were installed and Galaxy starts fine, but no tools
>> run. The error is always a variation of this:
>>
>> Could not find platform dependent libraries 
>> Consider setting $PYTHONHOME to [:]
>> Traceback (most recent call last):
>>   File
>> "galaxy-dist/database/jobs_directory/000/418/418014/set_meta
>> data_4LcsZv.py",
>> line 1, in 
>> from galaxy_ext.metadata.set_metadata import set_metadata;
>> set_metadata()
>>   File "galaxy-dist/lib/galaxy_ext/metadata/set_metadata.py", line 19,
>> in 
>> import json
>> ImportError: No module named json
>>
>> I put some debug output at the top of set_metadata.py and it is indeed
>> running the correct python version from the virtual env. Needless to say
>> I can import json just fine using that python version. Any ideas?
>>
>> Any help is greatly appreciated.
>>
>> Thanks
>> Ulf
>>
>>
> **
> The information contained in the EMail and any attachments is confidential
> and intended solely and for the attention and use of the named
> addressee(s). It may not be disclosed to any other person without the
> express authority of Public Health England, or the intended recipient, or
> both. If you are not the intended recipient, you must not disclose, copy,
> distribute or retain this message or any part of it. This footnote also
> confirms that this EMail has been swept for computer viruses by
> Symantec.Cloud, but please re-sweep any attachments before opening or
> saving. http://www.gov.uk/PHE
> **
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] error despite success

[Marius van den Beek]

Hi Matthias,

any chance https://github.com/galaxyproject/galaxy/pull/4644
would fix this for you ?

Best,
Marius

On 16 October 2017 at 20:12, Matthias Bernt  wrote:

> Dear list,
>
> today I got an erroneous result (red) despite the output of the tool to be
> seems fine. Seemingly there was an error during setting the metadata.
>
> ```
> Fatal error: An error occured during execution, see stderr and stdout for
> more information
> Reading reference gene model /gpfs1/data/galaxy_server/gala
> xy-dev/database/files/002/dataset_2992.dat ... Done
> Loading SAM/BAM file ...  Finished
> Total 62188 usable reads were sampled
> Traceback (most recent call last):
>   File 
> "/gpfs1/data/galaxy_server/galaxy-dev/jobs_dir/001/1858/set_metadata_Ad6q_v.py",
> line 1, in
> from galaxy_ext.metadata.set_metadata import set_metadata;
> set_metadata()
>   File 
> "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy_ext/metadata/set_metadata.py",
> line 85, in set_metadata
> datatypes_registry.load_datatypes(root_dir=galaxy_root,
> config=datatypes_config)
>   File 
> "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/datatypes/registry.py",
> line 109, in load_datatypes
> tree = galaxy.util.parse_xml(config)
>   File "/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/util/__init__.py",
> line 214, in parse_xml
> root = tree.parse(fname, parser=ElementTree.XMLParser(t
> arget=DoctypeSafeCallbackTarget()))
>   File 
> "/global/apps/bioinf/galaxy/bin/Python-2.7.13/lib/python2.7/xml/etree/ElementTree.py",
> line 647, in parse
> source = open(source, "rb")
> IOError: [Errno 2] No such file or directory:
> '/gpfs1/data/galaxy_server/galaxy-dev/database/tmp/tmp_uHZbW'
> ```
>
> Are there any suggestions how to continue with such a behavior. The file
> system on our cluster seems to be slow at the moment, could this be the
> cause?
>
> Best,
> Matthias
>
>
> --
>
> ---
> Matthias Bernt
> Bioinformatics Service
> Molekulare Systembiologie (MOLSYB)
> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
> Helmholtz Centre for Environmental Research GmbH - UFZ
> Permoserstraße 15, 04318 Leipzig, Germany
> 
> Phone +49 341 235 482296,
> m.be...@ufz.de, www.ufz.de
>
> Sitz der Gesellschaft/Registered Office: Leipzig
> Registergericht/Registration Office: Amtsgericht Leipzig
> Handelsregister Nr./Trade Register Nr.: B 4703
> Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig
> Wilfried Kraus
> Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
> Prof. Dr. Dr. h.c. Georg Teutsch
> Administrative Geschäftsführerin/ Administrative Managing Director:
> Prof. Dr. Heike Graßmann
> ---
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Show history structure : Error 'Undefined' object is not callable

[Marius van den Beek]

Hi Julie,

thanks for reporting this, this is indeed broken.
I hope https://github.com/galaxyproject/galaxy/pull/4794 is going to fix
this.

Best,
Marius

On 12 October 2017 at 14:53, julie dubois  wrote:

> Hi all,
> I'm on 17.05 galaxy version and display it on firefox.
> I'm trying to use "show structure" functionnality on my histories
> (I've never tried before this version) and I obtain this error :
>
>
> URL: http://192.168.100.98:8080/galaxy/history/display_structured
> File '/data/galaxy-dist/.venv/local/lib/python2.7/site-packages/weberror/
> evalexception.py',
> line 431 in respond
>   app_iter = self.application(environ, detect_start_response)
> File '/data/galaxy-dist/.venv/local/lib/python2.7/site-
> packages/paste/recursive.py',
> line 85 in __call__
>   return self.application(environ, start_response)
> File '/data/galaxy-dist/.venv/local/lib/python2.7/site-
> packages/paste/httpexceptions.py',
> line 640 in __call__
>   return self.application(environ, start_response)
> File '/data/galaxy-dist/lib/galaxy/web/framework/base.py', line 135 in
> __call__
>   return self.handle_request( environ, start_response )
> File '/data/galaxy-dist/lib/galaxy/web/framework/base.py', line 214 in
> handle_request
>   body = method( trans, **kwargs )
> File '/data/galaxy-dist/lib/galaxy/webapps/galaxy/controllers/history.py',
> line 571 in display_structured
>   return trans.fill_template( "history/display_structured.mako",
> items=items, history=history )
> File '/data/galaxy-dist/lib/galaxy/web/framework/webapp.py', line 891
> in fill_template
>   return self.fill_template_mako( filename, **kwargs )
> File '/data/galaxy-dist/lib/galaxy/web/framework/webapp.py', line 905
> in fill_template_mako
>   return template.render( **data )
> File '/data/galaxy-dist/.venv/local/lib/python2.7/site-
> packages/mako/template.py',
> line 445 in render
>   return runtime._render(self, self.callable_, args, data)
> File '/data/galaxy-dist/.venv/local/lib/python2.7/site-
> packages/mako/runtime.py',
> line 829 in _render
>   **_kwargs_for_callable(callable_, data))
> File '/data/galaxy-dist/.venv/local/lib/python2.7/site-
> packages/mako/runtime.py',
> line 864 in _render_context
>   _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
> File '/data/galaxy-dist/.venv/local/lib/python2.7/site-
> packages/mako/runtime.py',
> line 890 in _exec_template
>   callable_(context, *args, **kwargs)
> File '/data/galaxy-dist/database/compiled_templates/base.mako.py',
> line 63 in render_body
>   __M_writer(unicode(next.body()))
> File '/data/galaxy-dist/database/compiled_templates/history/dis
> play_structured.mako.py',
> line 50 in render_body
>   __M_writer(unicode(render_item( entity, children )))
> File '/data/galaxy-dist/database/compiled_templates/history/dis
> play_structured.mako.py',
> line 35 in render_item
>   return render_render_item(context._locals(__M_locals),entity,children)
> File '/data/galaxy-dist/database/compiled_templates/history/dis
> play_structured.mako.py',
> line 191 in render_render_item
>   render_item_job( entity, children )
> File '/data/galaxy-dist/database/compiled_templates/history/dis
> play_structured.mako.py',
> line 183 in render_item_job
>   return render_render_item_job(context,job,children)
> File '/data/galaxy-dist/database/compiled_templates/history/dis
> play_structured.mako.py',
> line 239 in render_render_item_job
>   __M_writer(unicode( show_params.inputs_recursive( tool.inputs,
> params_object, depth=1 ) ))
> File '/data/galaxy-dist/database/compiled_templates/show_params.mako.py',
> line 293 in render_inputs_recursive
>   for i, element in enumerate(listify(param_values[input.name])):
> TypeError: 'Undefined' object is not callable
>
>
> Is it a problem with this history (I've tested with a full green
> history and with history containing red jobs without difference in
> error message) or with my galaxy configuration may be ?
>
> Thanks.
>
>
> Julie
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Worklist failing on collection with AttributeError: 'object' object has no attribute 'collection'

[Marius van den Beek]

Hi Tony,

I was able to reproduce the problem, there is indeed a problem.
We're working on fixing this, but you should be able to include a separate
input step
before starting your workflow (In the left panel, under inputs -> Input
dataset collection).
Make sure to set the Collection type of this input step to "Paired", and
hopefully your workflow should run successfully.

Let me know if that works for you,
Marius

On 4 October 2017 at 15:02, Marius van den Beek <m.vandenb...@gmail.com>
wrote:

> Hi Tony,
>
> could you include the traceback that leads up to this error ?
> That should help us narrow down the cause.
>
> Best,
> Marius
>
> On 3 October 2017 at 18:37, Brooks, Tony <a.bro...@ucl.ac.uk> wrote:
>
>> Hello all
>>
>>
>> I am trying to invoke a workflow on a paired dataset collection (simple
>> trim step (trimmomatic) followed by align (RNAStar)) through the GUI.
>>
>> Each time I run, I get the prompt:
>>
>>
>>
>> Successfully invoked workflow *RNASeq v1.0*.
>>
>> You can check the status of queued jobs and view the resulting data by
>> refreshing the History pane. When the job has been run the status will
>> change from 'running' to 'finished' if completed successfully or 'error' if
>> problems were encountered
>>
>>
>>
>> suggesting that all is well, but nothing gets added to my history. I am
>> not sending data to another history, but to the current one.
>>
>>
>>
>> If I look into the logs, I see the error AttributeError: 'object' object
>> has no attribute 'collection' as the reason for failing
>>
>>
>>
>> If I modify the workflow to work on individual datasets, it runs
>> successfully. It seems to be purely when I run on collections.
>>
>>
>>
>> This is my own Galaxy instance (17.05), not the public server.
>>
>>
>> Any help appreciated
>>
>>
>>
>> Tony (Galaxy Newbie)
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Toolshed partial installation ? deeptools error

[Marius van den Beek]

Also upon re-reading I see that your dependency is being picked up from
/usr/local/bin,
which indicates that this isn't using conda at all.
A small excerpt from the logs when you try to run the tool may be helpful
to understand what is going on there.

Best,
Marius

On 3 October 2017 at 09:29, Marius van den Beek <m.vandenb...@gmail.com>
wrote:

> Hi Julie,
>
> any chance you are setting PYTHONPATH before starting galaxy's job
> handlers (a common place would the supervisor config, in case you're using
> that)?
> This will interfere with all python modules that are installed via conda.
>
> Best,
> Marius
>
> On 3 October 2017 at 09:13, julie dubois <dubju...@gmail.com> wrote:
>
>> Dear Greg,
>> I read your discussion with @bgruening and I think it's not exactly
>> the same. Because in my case, The executable IS present in my
>> galaxy-dist/tool_dependency/_conda/envs/__deeptools@2.5.1/bin .
>> The problem is far away in the execution when the tool (computeMatrix
>> in my example) try to load the python module deeptools.computeMatrix.
>> So is there a solution to discover where is the problem ?
>>
>> Thanks
>>
>> 2017-10-02 15:16 GMT+02:00 Von Kuster, Greg <g...@psu.edu>:
>> > This looks like the issue I’ve recently seen as well - I just discussed
>> this
>> > with @bgruening on gitter - https://gitter.im/galaxy-iuc/iuc.  I’m
>> using a
>> > work-around of creating the __deeptools conda env manually.  In my
>> case, I’m
>> > not installing the whole suite, just the bamCoverage tool from the TS -
>> > https://toolshed.g2.bx.psu.edu/view/bgruening/deeptools_bam_
>> coverage/5d11599b8a7d.
>> > My work-around is to create the conda env manually:
>> >
>> > $conda create -n __deeptools@2.5.1 deep tools=2.5.1
>> >
>> > This env is created in the conda environment outside of Galaxy, so I
>> just
>> > copy it to the conda environment I’ve configured for Galaxy.  This
>> works,
>> > but it seems there is some issue with the installation from the TS, so
>> > perhaps a more ling term fix is needed.
>> >
>> >
>> > On Oct 2, 2017, at 8:59 AM, julie dubois <dubju...@gmail.com> wrote:
>> >
>> > Hi dev team,
>> >
>> > I have a very big problem since I updated to galaxy 17.05 with
>> > deeptools from toolshed.
>> >
>> > Due to a bug in galaxy 17.05 with toolshed installed tools (reported
>> > in this issue https://github.com/galaxyproject/galaxy/issues/4591), I
>> > re-installed deeptools to try to resolve it.
>> > Not resolved really but it's another problem. I temporarly bypassed it.
>> >
>> > Here is my problem : I've installed deeptools from toolshed and when
>> > we try to use "compute matrix" for example, we have this error message
>> > :
>> >
>> > Fatal error: Exit code 1 ()
>> > Traceback (most recent call last):
>> >  File "/usr/local/bin/computeMatrix", line 4, in 
>> >from deeptools.computeMatrix import main
>> > ImportError: No module named deeptools.computeMatrix
>> >
>> > It seems that wrapper are installed but not the tool "deeptools".
>> >
>> > For more information, here it's the way for the installation :
>> > search and install via toolshed
>> > Install "suite_deeptools" (version 2.5.1.1.0 : 3d68b716965a)
>> > Info in Manage tool dependencies : version=2.5.1, resolver=Conda,
>> > green check symbol in the line with a list of all deeptools.
>> >
>> > Galaxy 17.05, commit cfabe37 (21 sept 2017)
>> >
>> > I've tried to uninstall/reinstall dependencies with conda but without
>> > success.
>> >
>> > Have you any idea to install the deeptools please ?
>> >
>> > Thanks.
>> > Julie
>> > ___
>> > Please keep all replies on the list by using "reply all"
>> > in your mail client.  To manage your subscriptions to this
>> > and other Galaxy lists, please use the interface at:
>> >  https://lists.galaxyproject.org/
>> >
>> > To search Galaxy mailing lists use the unified search at:
>> >  http://galaxyproject.org/search/
>> >
>> >
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Toolshed partial installation ? deeptools error

[Marius van den Beek]

Hi Julie,

any chance you are setting PYTHONPATH before starting galaxy's job handlers
(a common place would the supervisor config, in case you're using that)?
This will interfere with all python modules that are installed via conda.

Best,
Marius

On 3 October 2017 at 09:13, julie dubois  wrote:

> Dear Greg,
> I read your discussion with @bgruening and I think it's not exactly
> the same. Because in my case, The executable IS present in my
> galaxy-dist/tool_dependency/_conda/envs/__deeptools@2.5.1/bin .
> The problem is far away in the execution when the tool (computeMatrix
> in my example) try to load the python module deeptools.computeMatrix.
> So is there a solution to discover where is the problem ?
>
> Thanks
>
> 2017-10-02 15:16 GMT+02:00 Von Kuster, Greg :
> > This looks like the issue I’ve recently seen as well - I just discussed
> this
> > with @bgruening on gitter - https://gitter.im/galaxy-iuc/iuc.  I’m
> using a
> > work-around of creating the __deeptools conda env manually.  In my case,
> I’m
> > not installing the whole suite, just the bamCoverage tool from the TS -
> > https://toolshed.g2.bx.psu.edu/view/bgruening/deeptools_
> bam_coverage/5d11599b8a7d.
> > My work-around is to create the conda env manually:
> >
> > $conda create -n __deeptools@2.5.1 deep tools=2.5.1
> >
> > This env is created in the conda environment outside of Galaxy, so I just
> > copy it to the conda environment I’ve configured for Galaxy.  This works,
> > but it seems there is some issue with the installation from the TS, so
> > perhaps a more ling term fix is needed.
> >
> >
> > On Oct 2, 2017, at 8:59 AM, julie dubois  wrote:
> >
> > Hi dev team,
> >
> > I have a very big problem since I updated to galaxy 17.05 with
> > deeptools from toolshed.
> >
> > Due to a bug in galaxy 17.05 with toolshed installed tools (reported
> > in this issue https://github.com/galaxyproject/galaxy/issues/4591), I
> > re-installed deeptools to try to resolve it.
> > Not resolved really but it's another problem. I temporarly bypassed it.
> >
> > Here is my problem : I've installed deeptools from toolshed and when
> > we try to use "compute matrix" for example, we have this error message
> > :
> >
> > Fatal error: Exit code 1 ()
> > Traceback (most recent call last):
> >  File "/usr/local/bin/computeMatrix", line 4, in 
> >from deeptools.computeMatrix import main
> > ImportError: No module named deeptools.computeMatrix
> >
> > It seems that wrapper are installed but not the tool "deeptools".
> >
> > For more information, here it's the way for the installation :
> > search and install via toolshed
> > Install "suite_deeptools" (version 2.5.1.1.0 : 3d68b716965a)
> > Info in Manage tool dependencies : version=2.5.1, resolver=Conda,
> > green check symbol in the line with a list of all deeptools.
> >
> > Galaxy 17.05, commit cfabe37 (21 sept 2017)
> >
> > I've tried to uninstall/reinstall dependencies with conda but without
> > success.
> >
> > Have you any idea to install the deeptools please ?
> >
> > Thanks.
> > Julie
> > ___
> > Please keep all replies on the list by using "reply all"
> > in your mail client.  To manage your subscriptions to this
> > and other Galaxy lists, please use the interface at:
> >  https://lists.galaxyproject.org/
> >
> > To search Galaxy mailing lists use the unified search at:
> >  http://galaxyproject.org/search/
> >
> >
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Bug in "View all histories", history "disappears"

[Marius van den Beek]

Hello Christopher,

this is a longstanding bug that we recently fixed (initial description
https://github.com/galaxyproject/galaxy/issues/3519, fix
https://github.com/galaxyproject/galaxy/pull/4610 ),
This will be included in galaxy release 17.09, but the fix has also been
backported to relaese 16.10 and newer. If you update your galaxy instance
to the
latest commit this problem should be gone.

Best,
Marius

On 27 September 2017 at 09:26, Previti <
christopher.prev...@dkfz-heidelberg.de> wrote:

> Dear all,
>
> I noticed a bug in Galaxy (17.05), specifically when using "View all
> histories"  to switch between different histories.
> It happened to me multiple times that the history I switched away from was
> not shown anymore.
> Searching for the history by name made it show up again, but still not
> ideal.
> I have about 10 histories that I cycle through on a regular basis. Doesn't
> the system support that many?
>
> Cheers,
> Christopher
>
>
> --
> *Dr. Christopher Previti*
> Genomics and Proteomics Core Facility
> High Throughput Sequencing (W190)
> Bioinformatician
>
> German Cancer Research Center (DKFZ)
> Foundation under Public Law
> Im Neuenheimer Feld 580
> 69120 Heidelberg
> Germany
> Room: B2.102 (INF580/TP3)
> Phone: +49 6221 42-4434 <+49%206221%20424434>
>
> christopher.prev...@dkfz.de 
> www.dkfz.de
>
> Management Board: Prof. Dr. Michael Baumann, Prof. Dr. Josef Puchta
> VAT-ID No.: DE143293537
>
> Vertraulichkeitshinweis: Diese Nachricht ist ausschließlich für die
> Personen bestimmt, an die sie adressiert ist.
> Sie kann vertrauliche und/oder nur für den/die Empfänger bestimmte
> Informationen enthalten. Sollten Sie nicht
> der bestimmungsgemäße Empfänger sein, kontaktieren Sie bitte den Absender
> und löschen Sie die Mitteilung.
> Jegliche unbefugte Verwendung der Informationen in dieser Nachricht ist
> untersagt.
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Galaxy server requires a restart to activate tools

[Marius van den Beek]

Hi,

Can you check the commit of galaxy that you are using and post the logs
that you see when you install a tool that doesn't work ?
Does this happen for all tools that you install ?

Best,
Marius

On 19 September 2017 at 15:40, RMSe17  wrote:

> Hi Bjoern,
>
> I upgraded to 17.05, but am still getting the same error when trying
> to run an installed tool without Galaxy restart.
>
> Any idea on where the issue could be?
>
> Thanks!
>
> On 9/13/17, Björn Grüning  wrote:
> > Hi,
> >
> > afaik Marius fixed a few bugs in 17.05 and I know that 17.05 should work
> > as described not sure 17.01 can be exprected to work this way.
> >
> > Can you try 17.05?
> > Thanks,
> > Bjoern
> >
> > On 14.09.2017 03:51, RMSe17 wrote:
> >> Hello, I have a fresh 17.01 Galaxy installed and set up with uWSGI.
> >> After installing a new tool, I can't run it without first restarting
> >> Galaxy.  This is odd, because I know in 17.01 that should no longer be
> >> happening.
> >>
> >> The error I get is "This tool was disabled before job completed.
> >> Please contact your Galaxy administrator.
> >>
> >> In the logs I see "Tool (... tool name) removed from tool config,
> >> unable to run job
> >> Cleaning up external metadata files
> >> Error checking job readiness"
> >>
> >> After Galaxy restarts, the tool runs fine.
> >>
> >> Is there anything typical that could cause this?  Any place I should
> >> start looking?
> >>
> >> Thanks!
> >> ___
> >> Please keep all replies on the list by using "reply all"
> >> in your mail client.  To manage your subscriptions to this
> >> and other Galaxy lists, please use the interface at:
> >>   https://lists.galaxyproject.org/
> >>
> >> To search Galaxy mailing lists use the unified search at:
> >>   http://galaxyproject.org/search/
> >>
> >
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] installation/conda problems

[Marius van den Beek]

Hi Matthias,

Adding to Bjoern's suggestion you can also check in the admin section under
"Manage Tool Dependencies" into which path the dependency has been
installed (it may be mulled-),
and from there you can also uninstall the dependency.

You can also trigger uninstall of the conda dependency through the API (the
normal toolshed API does not uninstall conda dependencies),
an example for doing this can be found here:
https://github.com/galaxyproject/galaxy/blob/dev/test/integration/test_resolvers.py#L109
.
This hasn't made it into bioblend yet (but we should do that!).

Best,
Marius

On 5 September 2017 at 17:32, Björn Grüning 
wrote:

> Hi Matthias,
>
> please have a look into your  Do you see any `__intarna...` folder? If so try to remove this one.
>
> Cheers,
> Bjoern
>
> On 05.09.2017 17:21, Matthias Bernt wrote:
> > Dear list,
> >
> > I think I have messed up the environment for one tool (IntaRNA) -- can
> > someone tell me what to do to get a clear environment? Here is what
> > happened.
> >
> > I have started the installation (via ansible) and accidentally aborted
> > it. Then the tool gave me the following error:
> >
> > ```
> > Failed to activate conda environment! Error was: CondaEnvironmentError:
> > Environment error: Cannot activate environment bash. User does not have
> > write access for conda symlinks. Fatal error: Exit code 1 (Error
> > occurred. Please check Tool Standard Error)
> > ```
> >
> > Since uninstalling and reinstallation (via ansible) gave the same result
> > I tried to reinstall from the Galaxy UI. Now I got (even if if the admin
> > interface list the tool as installed):
> >
> > ```
> > Fatal error: Exit code 127 (Error occurred. Please check Tool Standard
> > Error)
> > /gpfs1/data/galaxy_server/galaxy-dev/jobs_dir/000/362/tool_script.sh:
> > line 25: IntaRNA: command not found
> > ```
> >
> > Is there a list of directories and files that I could clear to get rid
> > of any trace of the tool and start anew?
> >
> > Best,
> > Matthias
> >
> >
> >
> >
> >
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Watching the tools directory and dynamic reloading

[Marius van den Beek]

Hi Keith,

Galaxy has been able to load tools after installation for quite some time,
however that did not work for production setups that are using multiple
job handler threads.

We did indeed refine the tool loading to work for production servers
starting with release 16.10
(it's mentioned in the release notes
https://docs.galaxyproject.org/en/master/releases/16.10_announce.html).
Aspects of this have been refined up to release 17.05, which should be the
most reliable concerning this feature.

No special options need to be set to use this, it should hopefully just
work.
Technically we watch the various shed_tool_conf.xml / tool_conf.xml files
as well as the individual tool wrappers for changes
and trigger a reload, which we speed up by caching the tool definitions.
One thing we can't do at the moment is detect if a macro file that is
included in a tool has changes,
so in this case you'd still need to restart the galaxy instance
unfortunately.

There is another option (watch_tools, see
https://github.com/galaxyproject/galaxy/blob/release_17.05/config/galaxy.ini.sample#L257
)
 in the galaxy.ini that allows you to watch a directory of tools
(i.e no installation from the toolshed or modification of a tool_conf,xml
file necessary) that you can activate.
If you want to use this feature you need to indicate in your tool_conf.xml
file which directory you would like to monitor.
The syntax is like this:



(The recursive="true" is not necessary if you want to monitor a flat
directory).

Best,
Marius


On 20 August 2017 at 04:13, Keith Suderman  wrote:

> I thought I had read that newer versions of Galaxy were able to watch the
> tools directory and update the tools menu when any tool wrappers were
> added, deleted, or modified; basically the "Admin -> Reload a tool's
> configuration" without the administrator actually having to do anything.
> However, my Google searches are not turning up anything?  Am I
> over-remembering something I had read or are my Google skills failing me?
>
> This is for an upcoming course that will involve students developing tools
> and pipelines. We want the teaching assistants to be able to update their
> master Galaxy instance with new tools without a lot of button clicks or a
> server restart.
>
> Thanks,
> Keith
>
> --
> Keith Suderman
> Research Associate
> Department of Computer Science
> Vassar College, Poughkeepsie NY
> suder...@cs.vassar.edu
>
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Galaxy Conda resolver install Issue

[Marius van den Beek]

Hello Evan,

this looks suspiciously like a bug that Matthias Bernt has fixed on the
development branch a while back.
I have backported this to Galaxy release 17.01 and release 17.05.

If you update your galaxy branch to the latest commit it should work (or
you should receive a more instructive error message).

Hope that helps,
Marius

On 17 August 2017 at 18:32, evan clark  wrote:

> I just ran across an issue when attempting to load galaxy. I am not sure
> how to fix this issue or what could be causing it. Conda is installed
> properly.
>
>   File "/var/web_services/galaxy/lib/galaxy/webapps/galaxy/buildapp.py",
> line 58, in paste_app_factory
> app = galaxy.app.UniverseApplication( global_conf=global_conf,
> **kwargs )
>   File "/var/web_services/galaxy/lib/galaxy/app.py", line 113, in __init__
> self._configure_toolbox()
>   File "/var/web_services/galaxy/lib/galaxy/config.py", line 935, in
> _configure_toolbox
> self.toolbox = tools.ToolBox( tool_configs, self.config.tool_path,
> self )
>   File "/var/web_services/galaxy/lib/galaxy/tools/__init__.py", line 205,
> in __init__
> app=app,
>   File "/var/web_services/galaxy/lib/galaxy/tools/toolbox/base.py", line
> 1044, in __init__
> self._init_dependency_manager()
>   File "/var/web_services/galaxy/lib/galaxy/tools/toolbox/base.py", line
> 1057, in _init_dependency_manager
> self.dependency_manager = build_dependency_manager( self.app.config )
>   File "/var/web_services/galaxy/lib/galaxy/tools/deps/__init__.py", line
> 41, in build_dependency_manager
> dependency_manager = DependencyManager( **dependency_manager_kwds )
>   File "/var/web_services/galaxy/lib/galaxy/tools/deps/__init__.py", line
> 84, in __init__
> self.dependency_resolvers = self.__build_dependency_resolvers(
> conf_file )
>   File "/var/web_services/galaxy/lib/galaxy/tools/deps/__init__.py", line
> 193, in __build_dependency_resolvers
> return self.__default_dependency_resolvers()
>   File "/var/web_services/galaxy/lib/galaxy/tools/deps/__init__.py", line
> 201, in __default_dependency_resolvers
> CondaDependencyResolver(self),
>   File "/var/web_services/galaxy/lib/galaxy/tools/deps/resolvers/conda.py",
> line 134, in __init__
> self.disabled = not galaxy.tools.deps.installable.
> ensure_installed(conda_context, install_conda, self.auto_init)
>   File "/var/web_services/galaxy/lib/galaxy/tools/deps/installable.py",
> line 77, in ensure_installed
> return ensure_installed(installable_context, auto_init)
> TypeError: ensure_installed() takes exactly 3 arguments (2 given)
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] htseq-count/deseq2 conda dependency issues

[Marius van den Beek]

Hi Peter,

indeed the defaults channel should now be the lowest priority of all conda
channels.
This recommended setting in galaxy.ini is now

conda_ensure_channels = iuc,bioconda,conda-forge,defaults,r

Unfortunately these changes appear as changes are made within bioconda,
which usually
do not coincide with galaxy releases. That means that these settings may
need to be adjusted occasionally.
On a slightly unrelated note, if you have upgraded to galaxy 17.05 you
should also upgrade conda to 4.2.13 if you haven't done so (conda install
conda=4.2.13), for best experience.

Hope that helps,
Marius


On 17 August 2017 at 17:12, Peter Briggs 
wrote:

> Dear devs
>
> I've recently encountered some strange issues when running the htseq-count
> and deseq2 tools installed from the toolshed onto our production instance.
> In both cases there have been problems with a couple of the conda
> dependency installations.
>
> Specifically:
>
> - For htseq-count, runs failed with an error message about a missing
> libbz2.so.1.0.
>
> - For deseq2, runs failed silently first with the same "missing
> libbz2.so.1.0" message, then when this was fixed, with a new message about
> an undefined symbol in libreadline.so.6.
>
> In both cases I seemed to be able to fix the problems manually by
> activating the appropriate conda environments (under
> .../tool_dependencies/_conda/envs) and reinstalling the affected packages
> from different sources:
>
> - htseq-count: in __samtools@1.3.1, reinstalled bzip2 1.0.2 from
> 'conda-forge' rather than 'defaults'
>
> - deseq2: in __r-getopt@1.20.0, did the same for bzip2 and also got
> readline 6.3 from 'conda-forge' rather than 6.2 from 'defaults'
>
> However: deseq2 installed into a local Galaxy instance doesn't seem to
> have this problem, so I'm guessing it's something peculiar to the setup of
> the production instance.
>
> Can anyone suggest what might be happening here (e.g. some issue with the
> conda config for the production instance?), and how it might be avoided in
> future?
>
> Thanks for your help,
>
> Best wishes
>
> Peter
>
> (PS if it helps: the production instance is running Scientific Linux 6.8,
> and has recently been updated from v16.07 to v17.05. Both the tools were
> installed prior to the upgrade.)
>
> --
> Peter Briggs peter.bri...@manchester.ac.uk
> Bioinformatics Core Facility University of Manchester
> B.1083 Michael Smith Bldg Tel: (0161) 2751482
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Galaxy Job Error

[Marius van den Beek]

My understanding is that this is not an error, it's an interruption in the
communication with a client. This should be harmless if it doesn't happen
at a massive rate. You probably have many more of these in your logs.

On Jun 14, 2017 11:04 PM, "evan clark" <eclar...@fau.edu> wrote:

That may work, but now we are getting this error.

Exception happened during processing of request from ('127.0.0.1', 43030)
Traceback (most recent call last):
  File "/cm/shared/apps/galaxy/prod/galaxy/.venv/lib/python2.7/
site-packages/paste/httpserver.py", line 1085, in process_request_in_thread
self.finish_request(request, client_address)
  File 
"/cm/shared/apps/galaxy/requirements/python/lib/python2.7/SocketServer.py",
line 331, in finish_request
self.RequestHandlerClass(request, client_address, self)
  File 
"/cm/shared/apps/galaxy/requirements/python/lib/python2.7/SocketServer.py",
line 654, in __init__
self.finish()
  File 
"/cm/shared/apps/galaxy/requirements/python/lib/python2.7/SocketServer.py",
line 713, in finish
self.wfile.close()
  File "/cm/shared/apps/galaxy/requirements/python/lib/python2.7/socket.py",
line 283, in close
self.flush()
  File "/cm/shared/apps/galaxy/requirements/python/lib/python2.7/socket.py",
line 307, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe

Marius van den Beek <m.vandenb...@gmail.com>
June 14, 2017 4:31 PM
Hi Evan,

If this is something that you rarely see and your filesystem is on a
network drive,
you could try setting `retry_job_output_collection` to 10 in the galaxy.ini
file.
This will let galaxy try to collection the output files 10 times,
which should help if there is some delay for the filesystem to be updated.

Best,
Marius
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Galaxy Job Error

[Marius van den Beek]

Hi Evan,

If this is something that you rarely see and your filesystem is on a
network drive,
you could try setting `retry_job_output_collection` to 10 in the galaxy.ini
file.
This will let galaxy try to collection the output files 10 times,
which should help if there is some delay for the filesystem to be updated.

Best,
Marius

On 14 June 2017 at 22:02, evan clark  wrote:

> Has anyone seen a similar error like this before. We are unsure if galaxy
> is causing the issue or it is being cuased by slurm as it seems galaxy is
> prematurely deleting a file.
>
> galaxy.jobs.runners DEBUG 2017-06-14 19:36:45,719 (3261/143577) Unable to
> cleanup /cm/shared/apps/galaxy/prod/galaxy/database/jobs_directory/
> 003/3261/galaxy_3261.ec: [Errno 2] No such file or directory:
> '/cm/shared/apps/galaxy/prod/galaxy/database/jobs_directory/003/3261/
> galaxy_3261.ec'
> galaxy.jobs.output_checker DEBUG 2017-06-14 19:36:45,725 Tool produced
> standard error failing job - [slurmstepd: get_exit_code task 0 died by
> signal
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] conda installation fails

[Marius van den Beek]

That's a little weird, I don't think relying on the system's
LD_LIBRARY_PATH is good practice.
Maybe you could try what was suggested in the last answer here:
https://serverfault.com/questions/372987/centos-usr-local-lib-system-wide-ld-library-path
.

On 19 April 2017 at 17:57, Matthias Bernt  wrote:

> Dear list,
>
> Just found out that conda is not installing on our CentOS6.8 installation
> (might be related to earlier messages).
>
> The problem is that the shell script installing conda unsets the
> LD_LIBRARY_PATH which makes tar dysfunctional (tar is called by the
> installer).
>
> The command line that is executed during galaxy's installation is
>
> wget -q --recursive -O '/tmp/conda_install7tWo7r.sh' '
> https://repo.continuum.io/miniconda/Miniconda3-4.2.12-Linux-x86_64.sh' &&
> bash '/tmp/conda_install7tWo7r.sh' -b -p 
> '/gpfs1/data/galaxy_server/galaxy-dev/_conda'
> && /gpfs1/data/galaxy_server/galaxy-dev/_conda/bin/conda install -y -q
> conda=4.2.13
>
> Removing the unset LD_LIBRARY_PATH line (and fiddling around with some
> file size checks) is (seemingly) successful.
>
> There is an open issue on the conda github:
> https://github.com/conda/conda/issues/3632
>
> Any suggestions? The line removal procedure seems to be quite complicated
> in an automated ansible installation which I try to achieve.
>
> Cheers,
> Matthias
>
> --
>
> ---
> Matthias Bernt
> Bioinformatics Service
> Molekulare Systembiologie (MOLSYB)
> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
> Helmholtz Centre for Environmental Research GmbH - UFZ
> Permoserstraße 15, 04318 Leipzig, Germany
> Phone +49 341 235 482296,
> m.be...@ufz.de, www.ufz.de
>
> Sitz der Gesellschaft/Registered Office: Leipzig
> Registergericht/Registration Office: Amtsgericht Leipzig
> Handelsregister Nr./Trade Register Nr.: B 4703
> Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig
> Wilfried Kraus
> Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
> Prof. Dr. Dr. h.c. Georg Teutsch
> Administrative Geschäftsführerin/ Administrative Managing Director:
> Prof. Dr. Heike Graßmann
> ---
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] SSL certificate errors

[Marius van den Beek]

Hi Matthias,

this looks like your python installation does not know how to verify SSL
certificates.
Redhat have a nice section on troubleshooting that here:
https://access.redhat.com/articles/2039753

I can get your error by specifying a nonexsting cert:

mvandenb@Mariuss-MacBook-Pro:~/src/readtagger/readtagger (master)*$
SSL_CERT_FILE=/does_not_exist python ~/urllib2_test.py
https://usegalaxy.org
Traceback (most recent call last):
  File "/Users/mvandenb/urllib2_test.py", line 10, in 
urllib2.urlopen(req)
  File 
"/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 154, in urlopen
return opener.open(url, data, timeout)
  File 
"/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 429, in open
response = self._open(req, data)
  File 
"/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 447, in _open
'_open', req)
  File 
"/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 407, in _call_chain
result = func(*args)
  File 
"/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 1241, in https_open
context=self._context)
  File 
"/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 1198, in do_open
raise URLError(err)
urllib2.URLError: 

Hope that helps,
Marius
​

On 18 April 2017 at 17:13, Matthias Bernt  wrote:

> Dear list,
>
> Currently I get SSL certificate errors when I try to install a tool in
> Galaxy. I also get the same error when I try to use the galaxy-tools
> ansible role (when debugging this I thought this might be because my local
> server is on http, since the error appeared seemingly when a message was
> posted to the local server -- but I'm not entirely sure).
>
> I'm also missing the "Manage installed tools" link in the admin area (but
> maybe this is expected in a fresh install?).
>
> Thanks a lot.
>
> Best,
> Matthias
>
> This is the stack trace when I try to install through the web frontend:
>
> 141.65.27.89 - - [18/Apr/2017:14:58:19 +] "GET
> /admin_toolshed/prepare_for_install?changeset_revisions=9ba8
> ebb636f4_ids=d5dd1c5d2070513e_shed_url=https%3A%2F%
> 2Ftoolshed.g2.bx.psu.edu%2F HTTP/1.1" 500 - "-" "Mozilla/5.0 (X11;
> Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0"
>
> Error - :  CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)>
> URL: http://bioinf2-dev:8080/admin_toolshed/prepare_for_install?c
> hangeset_revisions=9ba8ebb636f4_ids=d5dd1c5d20705
> 13e_shed_url=https%3A%2F%2Ftoolshed.g2.bx.psu.edu%2F
> File 
> '/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/web/framework/middleware/error.py',
> line 154 in __call__
>   app_iter = self.application(environ, sr_checker)
> File '/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python2.7/
> site-packages/paste/recursive.py', line 85 in __call__
>   return self.application(environ, start_response)
> File '/gpfs1/data/galaxy_server/galaxy-dev/.venv/lib/python2.7/
> site-packages/paste/httpexceptions.py', line 640 in __call__
>   return self.application(environ, start_response)
> File '/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/web/framework/base.py',
> line 134 in __call__
>   return self.handle_request( environ, start_response )
> File '/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/web/framework/base.py',
> line 193 in handle_request
>   body = method( trans, **kwargs )
> File 
> '/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/web/framework/decorators.py',
> line 98 in decorator
>   return func( self, trans, *args, **kwargs )
> File '/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/webapps/gal
> axy/controllers/admin_toolshed.py', line 1035 in prepare_for_install
>   raw_text = util.url_get( tool_shed_url, 
> password_mgr=self.app.tool_shed_registry.url_auth(
> tool_shed_url ), pathspec=pathspec, params=params )
> File '/gpfs1/data/galaxy_server/galaxy-dev/lib/galaxy/util/__init__.py',
> line 1488 in url_get
>   response = urlopener.open( full_url )
> File '/global/apps/bioinf/galaxy/bin/Python-2.7.13/lib/python2.7/urllib2.py',
> line 429 in open
>   response = self._open(req, data)
> File '/global/apps/bioinf/galaxy/bin/Python-2.7.13/lib/python2.7/urllib2.py',
> line 447 in _open
>   '_open', req)
> File '/global/apps/bioinf/galaxy/bin/Python-2.7.13/lib/python2.7/urllib2.py',
> line 407 in _call_chain
>   result = func(*args)
> File '/global/apps/bioinf/galaxy/bin/Python-2.7.13/lib/python2.7/urllib2.py',
> line 1241 in https_open
>   context=self._context)
> File '/global/apps/bioinf/galaxy/bin/Python-2.7.13/lib/python2.7/urllib2.py',
> line 1198 in do_open
>   raise URLError(err)
> URLError:  verify failed (_ssl.c:661)>
>
>
> CGI Variables
> -
>   CONTENT_LENGTH: '0'
>   HTTP_ACCEPT: 'text/html,application/xhtml+x
> 

Re: [galaxy-dev] drmaa on univa grid engine

[Marius van den Beek]

Hi Matthias,

in addition to the link you posted there is also this here that looks
similar:
http://dev.list.galaxyproject.org/DRMAA-options-for-SGE-td4140931.html

Perhaps qsub is filling in a default queue, and drmaa is not doing that?
You could try adding a queue to the nativeSpecification.

Good luck,
Marius

On 13 April 2017 at 15:02, Matthias Bernt  wrote:

> Dear list,
>
> I'm struggling to get jobs running on our cluster (UGE).
>
> In the galaxy log I see:
>
> galaxy.jobs.runners.drmaa DEBUG 2017-04-13 12:24:57,799 (10) submitting
> file /gpfs1/data/galaxy_server/galaxy-dev/database/jobs_directory
> /000/10/galaxy_10.sh
>
> galaxy.jobs.runners.drmaa DEBUG 2017-04-13 12:24:57,799 (10)  native
> specification is: -l h_rt=60 -l h_vmem=1G -pe smp 2
>
> and I get
>
> galaxy.jobs.runners.drmaa WARNING 2017-04-13 12:24:57,805 (10)
> drmaa.Session.runJob() failed, will retry: code 17: error: no suitable
> queues
>
> But manual submission works:
>
> qsub -l h_rt=60 -l h_vmem=1G -pe smp 2 /gpfs1/data/galaxy_server/gala
> xy-dev/database/jobs_directory/000/10/galaxy_10.sh
> Your job 161150 ("galaxy_10.sh") has been submitted
>
> So I checked if I can use the drmaa library to submit using the following
> python script (based on the example2 included in the library):
>
> """
> #!/usr/bin/env python
>
> from __future__ import print_function
> import drmaa
> import os
>
>
> def main():
> """Submit a job.
> Note, need file called sleeper.sh in current directory.
> """
> s = drmaa.Session()
> s.initialize()
> print('Creating job template')
> jt = s.createJobTemplate()
> jt.remoteCommand = os.getcwd() + '/sleeper.sh'
> jt.args = ['42','Simon says:']
> jt.joinFiles=True
> jt.nativeSpecification = "-l h_rt=60 -l h_vmem=1G -pe smp 2"
>
> jobid = s.runJob(jt)
> print('Your job has been submitted with id ' + jobid)
>
> print('Cleaning up')
> s.deleteJobTemplate(jt)
> s.exit()
>
> if __name__=='__main__':
> main()
> """
>
> This gives me the same error:
>
> drmaa.errors.DeniedByDrmException: code 17: error: no suitable queues
>
> So it seems to be a problem related to the drmaa python library.
> Unfortunately I can not try without any native specs, since time, memory
> and cores need to be specifies (a suitable queue is chosen automatically).
>
> I found an earlier post with the same error (but this seemed to be caused
> by colons in the file name): http://dev.list.galaxyproject.
> org/Galaxy-with-Univa-Grid-Engine-UGE-instead-of-SGE-tt46578
> 48.html#a4657952
>
> Does anyone has suggestions for me?
>
> Best regards,
> Matthias
>
> --
>
> ---
> Matthias Bernt
> Bioinformatics Service
> Molekulare Systembiologie (MOLSYB)
> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
> Helmholtz Centre for Environmental Research GmbH - UFZ
> Permoserstraße 15, 04318 Leipzig, Germany
> Phone +49 341 235 482296,
> m.be...@ufz.de, www.ufz.de
>
> Sitz der Gesellschaft/Registered Office: Leipzig
> Registergericht/Registration Office: Amtsgericht Leipzig
> Handelsregister Nr./Trade Register Nr.: B 4703
> Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig
> Wilfried Kraus
> Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
> Prof. Dr. Dr. h.c. Georg Teutsch
> Administrative Geschäftsführerin/ Administrative Managing Director:
> Prof. Dr. Heike Graßmann
> ---
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] SLURM configuration problem

[Marius van den Beek]

Hello Leonor,

the log you've sent indicates that you're picking up pulsar from
/usr/local/lib.
That should not happen if you're running galaxy in a virtualenv.

Apart from that you did not mention if you able to submit slurm jobs from
the command line.
That is a prerequisite for launching jobs through galaxy.

Could you post the full startup logs and job_conf.xml file somewhere?

Best,
Marius

On 31 March 2017 at 12:38, Leonor Palmeira <lpalme...@chu.ulg.ac.be> wrote:

> Dear all,
>
> we are struggling with the basics in our Galaxy/SLURM configuration.
>
> - Galaxy is installed on a virtual machine that is physically
> independent from our cluster, but on a shared filesystem that is also
> mounted on the Cluster
>
> - Our Cluster is running SLURM and has 'slurm-drmaa' (Poznan version)
> installed. The shared filesystem is mounted on the same mount point as
> the VM, so their /paths are identical
>
> What do we need so that the Galaxy VM is able to submit jobs to the
> Cluster?
> Currently, running ".run.sh" from the VM as root leads to the $SGE_ROOT
> error I posted in my previous email and that ends like this :
>
>   File
> "/usr/local/lib/python2.7/dist-packages/pulsar/managers/
> util/drmaa/__init__.py",
> line 49, in __init__
> DrmaaSession.session.initialize()
>   File "/usr/local/lib/python2.7/dist-packages/drmaa/session.py", line
> 257, in initialize
> py_drmaa_init(contactString)
>   File "/usr/local/lib/python2.7/dist-packages/drmaa/wrappers.py", line
> 73, in py_drmaa_init
> return _lib.drmaa_init(contact, error_buffer, sizeof(error_buffer))
>   File "/usr/local/lib/python2.7/dist-packages/drmaa/errors.py", line
> 151, in error_check
> raise _ERRORS[code - 1](error_string)
> InternalException: code 1: Please set the environment variable SGE_ROOT.
>
> Any help would be greatly appreciated.
>
> Best
> Leonor
>
> Leonor Palmeira | PhD
> Associate Scientist
> Department of Human Genetics
> CHU de Liège | Domaine Universitaire du Sart-Tilman
> 4000 Liège | BELGIQUE
> Tél: +32-4-366.91.41
> Fax: +32-4-366.72.61
> e-mail: lpalme...@chu.ulg.ac.be
>
> On 02/20/2017 03:57 PM, Marius van den Beek wrote:
> > It doesn't hurt to try this, but I don't think that will solve the
> problem.
> >
> > Just to be sure, the basics are working? You can submit jobs via sbatch?
> > How did you compile/install slurm-drmaa ?
> >
> > Also it looks like drmaa-python is being used from /usr/local/... .
> > Are you running galaxy in a virtualenv?
> > It's strongly recommended to do that.
> > Starting galaxy through run.sh will handle the creation and installation
> > of all necessary dependencies for you.
> > Finally it looks like you're loading pulsar from /usr/local ... this is
> > a bit messy.
> > Please try getting the cluster submission to work using run.sh first.
> >
> >
> > On 20 February 2017 at 15:24, Leonor Palmeira <lpalme...@chu.ulg.ac.be
> > <mailto:lpalme...@chu.ulg.ac.be>> wrote:
> >
> > Hi Marius,
> >
> > yes, we are using the one from Poznan. Should we give it a try with
> the
> > fork?
> >
> > Best
> > Leonor
> >
> > Leonor Palmeira | PhD
> > Associate Scientist
> > Department of Human Genetics
> > CHU de Liège | Domaine Universitaire du Sart-Tilman
> > 4000 Liège | BELGIQUE
> > Tél: +32-4-366.91.41 <tel:%2B32-4-366.91.41>
> > Fax: +32-4-366.72.61 <tel:%2B32-4-366.72.61>
> > e-mail: lpalme...@chu.ulg.ac.be <mailto:lpalme...@chu.ulg.ac.be>
> >
> > On 02/20/2017 03:13 PM, Marius van den Beek wrote:
> > > Hi Leonor,
> > >
> > > Are you sure that you are using a drmaa library that is compatible
> with
> > > slurm?
> > > This http://apps.man.poznan.pl/trac/slurm-drmaa
> > <http://apps.man.poznan.pl/trac/slurm-drmaa> should work, IIRC,
> > > or alternatively you can use Nate Coraor's fork
> > > here https://github.com/natefoo/slurm-drmaa
> > <https://github.com/natefoo/slurm-drmaa>.
> > >
> > > Best,
> > > Marius
> > >
> > > On 20 February 2017 at 15:06, Leonor Palmeira <
> lpalme...@chu.ulg.ac.be <mailto:lpalme...@chu.ulg.ac.be>
> > > <mailto:lpalme...@chu.ulg.ac.be <mailto:lpalme...@chu.ulg.ac.be>>>
> > wrote:
> > >
> > > Hi,
> > >
> > > we modified our configuration as Marius suggested, but we
> 

Re: [galaxy-dev] Dataset Generation Errors

[Marius van den Beek]

Hi David,

I'm afraid I can't help you with this tool, but there is a metaphlan2
wrapper on the toolshed
that is developed and tested by the iuc (intergalactic utility commission,
https://galaxyproject.org/iuc/), you could give this a try.
I'm pretty sure this tool is functional.
In general galaxy will attempt to manage the tool dependencies for you. If
possible you should
let galaxy handle the tool dependency installation and avoid using
system-installed binaries.
You can find more information about the tool installation process here:
https://galaxyproject.org/admin/tools/add-tool-from-toolshed-tutorial/

Best,
Marius

On 24 March 2017 at 18:45, Lapointe, David <david.lapoi...@tufts.edu> wrote:

> Thanks for that tip. I did restart galaxy and that problem went away, but..
>
> bowtie2 is on the system path, but I am getting an error about bowtie2 not
> being found, even though the path is given to the program in the call (see
> below). Is this some weird config problem? Maybe try a different version.
> Tool: MetaPhlAn
> Name: MetaPhlAn on data 20 (rel_ab analysis)
> Created: Fri Mar 24 15:10:47 2017 (UTC)
> Filesize: 0 bytes
> Dbkey: ?
> Format: tabular
> Galaxy Tool ID: toolshed.g2.bx.psu.edu/repos/dannon/metaphlan/metaphlan/1.
> 7.0
> Galaxy Tool Version: 1.7.0
> Tool Version:
> Tool Standard Output: stdout
> <https://galaxy.tufts.edu/datasets/75732d013df69f30/stdout>
> Tool Standard Error: stderr
> <https://galaxy.tufts.edu/datasets/75732d013df69f30/stderr>
> Tool Exit Code: 1
> History Content API ID: 75732d013df69f30
> Job API ID: 46c16c6e54962e29
> History API ID: 7ca8f1b7f24e5a2d
> UUID: 42cc9323-4489-48f0-b419-72ce770925cb
> Full Path: /mnt/galaxy/files/000/dataset_709.dat
> Job Command-Line: python ${METAPHLAN_PATH}/metaphlan.py
> /mnt/galaxy/files/000/dataset_700.dat -t rel_ab --tax_lev a --bowtie2_exe
> /usr/bin/bowtie2 --bowtie2db ${METAPHLAN_PATH}/bowtie2db/mpa --no_map
> --bt2_ps sensitive-local -o /mnt/galaxy/files/000/dataset_709.dat
> Job Runtime (Wall Clock) 9 seconds
> Cores Allocated 1
> Job Start Time 2017-03-24 15:10:49
> Job End Time 2017-03-24 15:10:58
> Input Parameter Value Note for rerun
> Input metagenome (multi-fasta of metagenomic reads, loaded with the Get
> Data module, see below for an example) http://huttenhower.sph.
> harvard.edu/sites/default/files/LC1.fna
> <https://galaxy.tufts.edu/datasets/4f005d042f4be528/show_params>
> Type of analysis to perform rel_ab
> Taxonomic level all taxonomic levels
> Sensitivity options for read-marker similarity (as described by BowTie2) 
> Sensitive
> Local
> --
> David Lapointe Ph.D.
> Sr. Bioinformatics Specialist
> Research Technology (RT)
> Tufts Technology Services (TTS)
> 16 Dearborn Road
> Somerville MA 02144
>
> Phone:  617-627-5319
> Fax: 617-627-3667
> http://it.tufts.edu
>
> From: Marius van den Beek <m.vandenb...@gmail.com>
> Date: Thursday, March 23, 2017 at 11:13 AM
> To: David Lapointe <david.lapoi...@tufts.edu>
> Cc: "galaxy-dev@lists.galaxyproject.org" <galaxy-dev@lists.
> galaxyproject.org>
> Subject: Re: [galaxy-dev] Dataset Generation Errors
>
> Hi David,
>
> this typically happens on older Galaxy versions (< 17.01) after installing
> new or updated tools.
> A galaxy restart should help.
>
> Best,
> Marius
>
> On 23 March 2017 at 15:57, Lapointe, David <david.lapoi...@tufts.edu>
> wrote:
>
>> I have installed the metaphlan tool ( dannon e951f9d38339) which
>> installed with the dependancies seemingly satisfied. I am getting an error
>> "This tool was disabled before the job completed"  I can't determine why
>> this happens.  In the handler log is this:
>>
>> Tool 'toolshed.g2.bx.psu.edu/repos/dannon/metaphlan/metaphlan/1.7.0'
>> removed from tool config, unable to run job
>>
>>
>> David
>> --
>> David Lapointe Ph.D.
>> Sr. Bioinformatics Specialist
>> Research Technology (RT)
>> Tufts Technology Services (TTS)
>> 16 Dearborn Road
>> Somerville MA 02144
>>
>> Phone:  617-627-5319
>> Fax: 617-627-3667
>> http://it.tufts.edu
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] Dataset Generation Errors

[Marius van den Beek]

Hi David,

this typically happens on older Galaxy versions (< 17.01) after installing
new or updated tools.
A galaxy restart should help.

Best,
Marius

On 23 March 2017 at 15:57, Lapointe, David  wrote:

> I have installed the metaphlan tool ( dannon e951f9d38339) which installed
> with the dependancies seemingly satisfied. I am getting an error
> "This tool was disabled before the job completed"  I can't determine why
> this happens.  In the handler log is this:
>
> Tool 'toolshed.g2.bx.psu.edu/repos/dannon/metaphlan/metaphlan/1.7.0'
> removed from tool config, unable to run job
>
>
> David
> --
> David Lapointe Ph.D.
> Sr. Bioinformatics Specialist
> Research Technology (RT)
> Tufts Technology Services (TTS)
> 16 Dearborn Road
> Somerville MA 02144
>
> Phone:  617-627-5319
> Fax: 617-627-3667
> http://it.tufts.edu
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/

Re: [galaxy-dev] startup bug in lib/galaxy/tools/toolbox/base.py

[Marius van den Beek]

Hi Matthias,

so typically what's happening there is that there is no
tool_shed_respository entry in the database for this tool.
I see this in my development instance when I setup a new database, but keep
my old shed_tool_conf.xml.

If you want to get rid of these errors you can uninstall the tools in
question and install them again from the toolshed.

It could be worth thinking about relaxing the assumption that tools in the
shed_tool_conf.xml should have an entry in the database.
I'd encourage you to open an issue on
https://github.com/galaxyproject/galaxy/issues to get some more opinions on
this.


Best,
Marius

On 16 March 2017 at 09:33, Matthias Bernt  wrote:

> Dear galaxy developers,
>
> I sent this originally to the galaxy-bugs list and was asked to forward
> this also to this list. What is the best way to report bugs? There is
> galaxy-bugs, galaxy-dev and also the issue tracker on galaxy.
>
> During startup of my galaxy 17.01 installation an error message pops up:
>
> galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,605 Error reading tool
> from path: toolshed.g2.bx.psu.edu/repos/devteam/bwa/53646fef/bwa/bw
> a.xml
> Traceback (most recent call last):
>   File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560,
> in _load_tool_tag_set
> tool.tool_shed = tool_shed_repository.tool_shed
> AttributeError: 'NoneType' object has no attribute 'tool_shed'
>
> Seems that tool_shed_repository is still None at this point. This happens
> for some tools.
>
> galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,764 Error reading tool
> from path: toolshed.g2.bx.psu.edu/repos/devteam/bwa/53646fef/bwa/bw
> a-mem.xml
> Traceback (most recent call last):
>   File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560,
> in _load_tool_tag_set
> tool.tool_shed = tool_shed_repository.tool_shed
> AttributeError: 'NoneType' object has no attribute 'tool_shed'
> galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,768 Error reading tool
> from path: toolshed.g2.bx.psu.edu/repos/devteam/fastqc/a00a6402d09a/fas
> tqc/rgFastQC.xml
> Traceback (most recent call last):
>   File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560,
> in _load_tool_tag_set
> tool.tool_shed = tool_shed_repository.tool_shed
> AttributeError: 'NoneType' object has no attribute 'tool_shed'
> galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,773 Error reading tool
> from path: toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_qualit
> y/25c24379693a/fastq_trimmer_by_quality/fastq_trimmer_by_quality.xml
> Traceback (most recent call last):
>   File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560,
> in _load_tool_tag_set
> tool.tool_shed = tool_shed_repository.tool_shed
> AttributeError: 'NoneType' object has no attribute 'tool_shed'
> galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,779 Error reading tool
> from path: toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer_by_qualit
> y/04a609b0fdf6/fastq_trimmer_by_quality/fastq_trimmer_by_quality.xml
> Traceback (most recent call last):
>   File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560,
> in _load_tool_tag_set
> tool.tool_shed = tool_shed_repository.tool_shed
> AttributeError: 'NoneType' object has no attribute 'tool_shed'
> galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,788 Error reading tool
> from path: toolshed.g2.bx.psu.edu/repos/devteam/fastq_trimmer/e0cfb5a70
> 3ce/fastq_trimmer/fastq_trimmer.xml
> Traceback (most recent call last):
>   File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560,
> in _load_tool_tag_set
> tool.tool_shed = tool_shed_repository.tool_shed
> AttributeError: 'NoneType' object has no attribute 'tool_shed'
> galaxy.tools.toolbox.base ERROR 2017-03-14 11:15:17,793 Error reading tool
> from path: toolshed.g2.bx.psu.edu/repos/geert-vandeweyer/fastq_qc_trimm
> er/cba6282b5dc8/fastq_qc_trimmer/Paired_fastQ_trimmer.xml
> Traceback (most recent call last):
>   File "/home/berntm/galaxy/lib/galaxy/tools/toolbox/base.py", line 560,
> in _load_tool_tag_set
> tool.tool_shed = tool_shed_repository.tool_shed
> AttributeError: 'NoneType' object has no attribute 'tool_shed'
>
> Best,
> Matthias
>
> --
>
> ---
> Matthias Bernt
> Bioinformatics Service
> Molekulare Systembiologie (MOLSYB)
> Helmholtz-Zentrum für Umweltforschung GmbH - UFZ/
> Helmholtz Centre for Environmental Research GmbH - UFZ
> Permoserstraße 15, 04318 Leipzig, Germany
> Phone +49 341 235 482296,
> m.be...@ufz.de, www.ufz.de
>
> Sitz der Gesellschaft/Registered Office: Leipzig
> Registergericht/Registration Office: Amtsgericht Leipzig
> Handelsregister Nr./Trade Register Nr.: B 4703
> Vorsitzender des Aufsichtsrats/Chairman of the Supervisory Board: MinDirig
> Wilfried Kraus
> Wissenschaftlicher Geschäftsführer/Scientific Managing Director:
> Prof. Dr. Dr. h.c. Georg Teutsch
> Administrative Geschäftsführerin/ Administrative 

Re: [galaxy-dev] Problem with installing tools using Conda

[Marius van den Beek]

Yep:

PYTHONHOME=~/.venv conda --help
Traceback (most recent call last):
  File "/home/marius/conda/bin/conda", line 3, in 
from conda.cli import main
ImportError: No module named conda.cli

​

On 3 March 2017 at 11:04, Ashok Varadharajan <ashv...@gmail.com> wrote:

> Hi Marius,
>
> Thanks for your reply.
>
> Below is my supervisor configuration and it looks similar to yours except
> "PYTHONHOME" in the environment section. Do you think that would be the
> issue ?
>
> [supervisord]
>
> [program:galaxy_web]
> command = $GALAXY_PATH/.venv/bin/uwsgi --virtualenv
> $GALAXY_PATH/.venv --ini-paste $GALAXY_PATH/config/galaxy.ini --logdate
> --master --processes 3 --threads 6 --logto $GALAXY_PATH/logs/galaxy.uwsgi.log
> --socket 127.0.0.1:4001 --pythonpath lib --stats 127.0.0.1:9191
> directory   = $GALAXY_PATH
> umask   = 022
> autostart   = true
> autorestart = true
> startsecs   = 20
> user= galaxy
> environment = PATH=$GALAXY_PATH/.venv/bin:/u
> sr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,
> *PYTHONHOME=$GALAXY_PATH/.venv*,DRMAA_LIBRARY_PATH=/usr/local/l
> ib/libdrmaa.so
> numprocs = 1
> stoplight = INT
> startretries= 15
> process_name= web%(process_num)s
>
> [program:handler]
> command = $GALAXY_PATH/.venv/bin/python ./lib/galaxy/main.py -c
> $GALAXY_PATH/config/galaxy.ini --server-name=handler%(process_num)s
> --log-file=$GALAXY_PATH/logs/
> >galaxy_handler%(process_num)s.log
> directory   = $GALAXY_PATH
> process_name= handler%(process_num)s
> numprocs = 3
> umask   = 022
> autostart   = true
> autorestart = true
> startsecs   = 20
> user= galaxy
> environment = PATH=$GALAXY_PATH/.venv/bin:/u
> sr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin,
> *PYTHONHOME=$GALAXY_PATH/.venv*,DRMAA_LIBRARY_PATH=/usr/local/l
> ib/libdrmaa.so
> startretries= 15
>
>
> Thanks
> Ashok
>
>
>
> On Fri, Mar 3, 2017 at 10:37 AM, Marius van den Beek <
> m.vandenb...@gmail.com> wrote:
>
>> Hi Ashok,
>>
>> You should not set the PYTHONPATH in supervisor.
>> That is indeed going to conflict with conda.
>> If you are using uwsgi you can use something like this:
>>
>> [program:galaxy_web]
>> command = /bioinfo/guests/mvandenb/galaxy/.venv/bin/uwsgi 
>> --virtualenv /bioinfo/guests/mvandenb/galaxy/.venv --ini-paste 
>> /bioinfo/guests/mvandenb/gx/config/galaxy.ini --logdate
>>  --master --processes 2 --threads 2 --logto /home/galaxy/galaxy/uwsgi.log 
>> --socket 127.0.0.1:4001 --pythonpath lib --stats 127.0.0.1:9191
>> directory   = /bioinfo/guests/mvandenb/galaxy
>> umask   = 022
>> autostart   = true
>> autorestart = true
>> startsecs   = 20
>> user= galaxy
>> environment = 
>> /bioinfo/guests/mvandenb/galaxy/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>> numprocs= 1
>> stopsignal  = INT
>> startretries= 15
>>
>> Best,
>> Marius
>> ​
>>
>> On 3 March 2017 at 10:29, Ashok Varadharajan <ashv...@gmail.com> wrote:
>>
>>> Thanks John and Bjoern for your reply.
>>>
>>> Conda is working properly when i trigger it manually as a galaxy user.
>>> (example: $conda_prefix/conda install bwa -c bioconda)
>>>
>>> I tried to remove the conda_prefix directory and it was reinstalled
>>> after restarting the galaxy. But still I am getting the same issue when
>>> installing tools from the toolshed.
>>>
>>> I also tried updating the conda using "conda update conda" but didnt
>>> help.
>>>
>>> Do i have to adjust the PYTHONPATH in my supervisor configuration ?
>>>
>>> Regards,
>>> Ashok
>>>
>>>
>>>
>>> On Thu, Mar 2, 2017 at 10:14 PM, Björn Grüning <
>>> bjoern.gruen...@gmail.com> wrote:
>>>
>>>> Hi Ashok,
>>>>
>>>> this seems to be a problem with your conda installation.
>>>>
>>>> See for example here: https://github.com/conda/conda/issues/2463
>>>> Can you update your conda manually?
>>>>
>>>> Cheers,
>>>> Bjoern
>>>>
>>>> Am 02.03.2017 um 17:47 schrieb Ashok Varadharajan:
>>>> > ImportError: No module named conda.cli.main
>>>>
>>>
>>>
>>>
>>> --
>>> *Thanks and Regards,*
>>> *Ashok Varadharajan*
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>   http://galaxyproject.org/search/mailinglists/
>>>
>>
>>
>
>
> --
> *Thanks and Regards,*
> *Ashok Varadharajan*
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Problem with installing tools using Conda

[Marius van den Beek]

Hi Ashok,

You should not set the PYTHONPATH in supervisor.
That is indeed going to conflict with conda.
If you are using uwsgi you can use something like this:

[program:galaxy_web]
command = /bioinfo/guests/mvandenb/galaxy/.venv/bin/uwsgi
--virtualenv /bioinfo/guests/mvandenb/galaxy/.venv --ini-paste
/bioinfo/guests/mvandenb/gx/config/galaxy.ini --logdate
 --master --processes 2 --threads 2 --logto
/home/galaxy/galaxy/uwsgi.log --socket 127.0.0.1:4001 --pythonpath lib
--stats 127.0.0.1:9191
directory   = /bioinfo/guests/mvandenb/galaxy
umask   = 022
autostart   = true
autorestart = true
startsecs   = 20
user= galaxy
environment =
/bioinfo/guests/mvandenb/galaxy/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
numprocs= 1
stopsignal  = INT
startretries= 15

Best,
Marius
​

On 3 March 2017 at 10:29, Ashok Varadharajan  wrote:

> Thanks John and Bjoern for your reply.
>
> Conda is working properly when i trigger it manually as a galaxy user.
> (example: $conda_prefix/conda install bwa -c bioconda)
>
> I tried to remove the conda_prefix directory and it was reinstalled after
> restarting the galaxy. But still I am getting the same issue when
> installing tools from the toolshed.
>
> I also tried updating the conda using "conda update conda" but didnt help.
>
> Do i have to adjust the PYTHONPATH in my supervisor configuration ?
>
> Regards,
> Ashok
>
>
>
> On Thu, Mar 2, 2017 at 10:14 PM, Björn Grüning 
> wrote:
>
>> Hi Ashok,
>>
>> this seems to be a problem with your conda installation.
>>
>> See for example here: https://github.com/conda/conda/issues/2463
>> Can you update your conda manually?
>>
>> Cheers,
>> Bjoern
>>
>> Am 02.03.2017 um 17:47 schrieb Ashok Varadharajan:
>> > ImportError: No module named conda.cli.main
>>
>
>
>
> --
> *Thanks and Regards,*
> *Ashok Varadharajan*
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] SLURM configuration problem

[Marius van den Beek]

It doesn't hurt to try this, but I don't think that will solve the problem.

Just to be sure, the basics are working? You can submit jobs via sbatch?
How did you compile/install slurm-drmaa ?

Also it looks like drmaa-python is being used from /usr/local/... .
Are you running galaxy in a virtualenv?
It's strongly recommended to do that.
Starting galaxy through run.sh will handle the creation and installation of
all necessary dependencies for you.
Finally it looks like you're loading pulsar from /usr/local ... this is a
bit messy.
Please try getting the cluster submission to work using run.sh first.


On 20 February 2017 at 15:24, Leonor Palmeira <lpalme...@chu.ulg.ac.be>
wrote:

> Hi Marius,
>
> yes, we are using the one from Poznan. Should we give it a try with the
> fork?
>
> Best
> Leonor
>
> Leonor Palmeira | PhD
> Associate Scientist
> Department of Human Genetics
> CHU de Liège | Domaine Universitaire du Sart-Tilman
> 4000 Liège | BELGIQUE
> Tél: +32-4-366.91.41
> Fax: +32-4-366.72.61
> e-mail: lpalme...@chu.ulg.ac.be
>
> On 02/20/2017 03:13 PM, Marius van den Beek wrote:
> > Hi Leonor,
> >
> > Are you sure that you are using a drmaa library that is compatible with
> > slurm?
> > This http://apps.man.poznan.pl/trac/slurm-drmaa should work, IIRC,
> > or alternatively you can use Nate Coraor's fork
> > here https://github.com/natefoo/slurm-drmaa.
> >
> > Best,
> > Marius
> >
> > On 20 February 2017 at 15:06, Leonor Palmeira <lpalme...@chu.ulg.ac.be
> > <mailto:lpalme...@chu.ulg.ac.be>> wrote:
> >
> > Hi,
> >
> > we modified our configuration as Marius suggested, but we still get
> the
> > following error. This is an error we had just before, and we were
> trying
> > to fix it by specifying an $SGE_ROOT variable.
> >
> > I don't know why this error pops up, as we are trying to use SLURM,
> not
> > SGE...
> >
> > galaxy.jobs.runners.state_handler_factory DEBUG 2017-02-20
> 14:58:59,768
> > Loaded 'failure' state handler from module
> > galaxy.jobs.runners.state_handlers.resubmit
> > pulsar.managers.util.drmaa DEBUG 2017-02-20 14:58:59,807
> > Initializing DRMAA
> > session from thread MainThread
> > Traceback (most recent call last):
> >   File
> > "/home/mass/GAL/APP/galaxy/lib/galaxy/webapps/galaxy/buildapp.py",
> > line 55, in paste_app_factory
> > app = galaxy.app.UniverseApplication( global_conf=global_conf,
> > **kwargs )
> >   File "/home/mass/GAL/APP/galaxy/lib/galaxy/app.py", line 170, in
> > __init__
> > self.job_manager = manager.JobManager( self )
> >   File "/home/mass/GAL/APP/galaxy/lib/galaxy/jobs/manager.py", line
> > 23, in
> > __init__
> > self.job_handler = handler.JobHandler( app )
> >   File "/home/mass/GAL/APP/galaxy/lib/galaxy/jobs/handler.py", line
> > 32, in
> > __init__
> > self.dispatcher = DefaultJobDispatcher( app )
> >   File "/home/mass/GAL/APP/galaxy/lib/galaxy/jobs/handler.py", line
> > 723, in
> > __init__
> > self.job_runners = self.app.job_config.get_job_runner_plugins(
> > self.app.config.server_name )
> >   File "/home/mass/GAL/APP/galaxy/lib/galaxy/jobs/__init__.py", line
> > 687, in
> > get_job_runner_plugins
> > rval[id] = runner_class( self.app, runner[ 'workers' ],
> > **runner.get(
> > 'kwds', {} ) )
> >   File "/home/mass/GAL/APP/galaxy/lib/galaxy/jobs/runners/drmaa.py",
> > line
> > 88, in __init__
> > self.ds = DrmaaSessionFactory().get()
> >   File
> > "/usr/local/lib/python2.7/dist-packages/pulsar/managers/
> util/drmaa/__init__.py",
> >
> > line 31, in get
> > return DrmaaSession(session_constructor, **kwds)
> >   File
> > "/usr/local/lib/python2.7/dist-packages/pulsar/managers/
> util/drmaa/__init__.py",
> >
> > line 49, in __init__
> > DrmaaSession.session.initialize()
> >   File "/usr/local/lib/python2.7/dist-packages/drmaa/session.py",
> > line 257,
> > in initialize
> > py_drmaa_init(contactString)
> >   File "/usr/local/lib/python2.7/dist-packages/drmaa/wrappers.py",
> > line 73,
> > in py_drmaa_init
> > return _lib.drmaa_init(contact, error_buffer,
> sizeof(error_buffer))
> >   File "/usr

Re: [galaxy-dev] installing tool dependencies

[Marius van den Beek]

Hello Jochen,

what version of galaxy are you running?
When you are doing this you should have a traceback in your logs,
this would be helpful to understand what is going on.

When you say "I installed a bunch of tools at the same time",
did you really trigger multiple installations at the same time,
or did you install a suite of tools?
I don't think the install framework can handle parallel installations
(well).
It may work once in a while, but you're better of installing things
sequentially.

Best,
Marius


On 12 January 2017 at 10:47, Jochen Bick  wrote:

> Hi,
>
> I installed a bunch of tools at the same time and ran into some errors.
> After repairing those tools some of the dependencies installations did
> not work. If I try to repair the tools again I also get this error message:
>
> Internal Server Error
> Galaxy was unable to successfully complete your request
>
> An error occurred.
> This may be an intermittent problem due to load or other unpredictable
> factors, reloading the page may address the problem.
>
> The error has been logged to our team.
>
>
> the tool I tried is called:
> Name:
> camera_annotate
> Description:
> [Metabolomics][W4M][LC-MS] CAMERA R Package - Annotation - Returns
> annotation results (isotope peaks, adducts and fragments)
>
> Maybe anyone has an idea how to solve this. Thanks in advance
>
>
> Cheers Jochen
>
>
> --
> ETH Zurich
> *Jochen Bick*
> Animal Physiology
> Institute of Agricultural Sciences
> Postal address: Universitätstrasse 2 / LFW B 58.1
> Office: Tannenstrasse 1 / TAN D 6.2
> 8092 Zurich, Switzerland
>
> Phone +41 44 632 28 25
> jochen.b...@usys.ethz.ch 
> www.ap.ethz.ch
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Pulsar: issue configuring to use shared filesystem rather than staging data

[Marius van den Beek]

I hope John can give some more input on this,

but if you're really short on time you could try the ssh runner

directly from galaxy without pulsar. I'm very happy with this driving a 
Torque 4 cluster.


The relevant part of my job_conf.xml looks like this:

```


SecureShell
Torque
myusername
submitnode.curie.fr
id="job_Resource_List">walltime=2:00:00,nodes=1:ppn=8,mem=32gb

true
file="/bioinfo/guests/mvandenb/galaxy/.venv/bin/activate" />




```

You'll only have to be able to do a passwordless ssh login using your 
galaxy user.


Best,

Marius


On 01/12/2016 15:35, Philip Blood wrote:

Hi Folks,

I'm working on a time-sensitive project (just a day or two left to 
sort it out) that requires I be able to submit jobs from a remote 
resource (TACC) to my resources (Pittsburgh Supercomputing Center) 
using a shared filesystem for data rather than staging via Pulsar. 
I've tried to set it up so that Pulsar does not do any staging, but is 
just handling the remote job submission and tool dependencies. 
However, Pulsar continues to use a local staging directory for the 
data and then copies the data back to the Galaxy directory.


What I'm hoping to do is have all the data stay in the shared 
filesystem for the entire course of the job. The quick test below does 
not have input data, but just issues commands on the remote compute 
node and generates output.


I'm using latest stable releases of Galaxy (16.07, installed on a TACC 
VM) and Pulsar (installed via pip at PSC). If anyone can provide some 
quick pointers I'd appreciate it.


Here are relevant parts of my configuration and some of the log output:

_Shared filesystem:_ /hopper

*_Local staging dir created by 
Pulsar:_* /usr/local/packages/pulsar/etc/files/staging


*_galaxy.ini_*
file_path = /hopper/sy3l67p/blood/galaxy_home/database/files
new_file_path = /hopper/sy3l67p/blood/galaxy_home/database/tmp
job_working_directory = 
/hopper/sy3l67p/blood/galaxy_home/database/job_working_directory
tool_dependency_dir = 
/hopper/sy3l67p/blood/galaxy_home/database/dependencies
dependency_resolvers_config_file = 
/hopper/sy3l67p/blood/galaxy_home/config/dependency_resolvers_conf.xml


*_job_conf.xml
_*
   
https://128.182.99.126:8913/
none
id="file_action_config">/home/tg455546/galaxy/config/file_actions.yaml

remote
-q batch



*_file_actions.yml_*
paths:
  # If Galaxy, the Pulsar, and the compute nodes all mount the same 
directory

  # staging can be disabled altogether for given paths.
  - path: /hopper/sy3l67p/blood/galaxy_home/database/files
action: none
  - path: /hopper/sy3l67p/blood/galaxy_home/database/job_working_directory
action: none

*_app.yml_*
---
manager:
  type: queued_drmaa
dependency_resolvers_config_file: 
/usr/local/packages/pulsar/etc/dependency_resolvers_conf.xml

tool_dependency_dir: /usr/local/packages/pulsar/dependencies

*_Galaxy paster.log_*
galaxy.jobs  DEBUG 2016-12-01 08:07:35,770 (50) 
Working directory for job is: 
/hopper/sy3l67p/blood/galaxy_home/database/job_working_directory\

/000/50
pulsar.client.staging.down INFO 2016-12-01 08:08:08,305 collecting 
output output with action FileAction[action_type=copy]
pulsar.client.client DEBUG 2016-12-01 08:08:08,814 Copying path 
[/usr/local/packages/pulsar/etc/files/staging/50/working/output] to 
[/hopper/\

sy3l67p/blood/galaxy_home/database/files/000/dataset_50.dat]

*_Pulsar uwsgi.log_*
2016-12-01 09:07:38,174 INFO 
 [pulsar.managers.base.base_drmaa][[manager=_default_]-[action=preprocess]-[job=50]] 
Submitting DRMA\

A job with nativeSpecification [-q batch]
t #78bc [   539.97] -> drmaa_allocate_job_template
t #78bc [   539.97] <- drmaa_allocate_job_template =0: jt=0x7f69cc002f80
t #78bc [   539.97] -> drmaa_set_attribute(jt=0x7f69cc002f80, 
name='drmaa_remote_command', value='/usr/local/packages/pulsar/etc/\

files/staging/50/command.sh')
t #78bc [   539.97] -> 
fsd_template_set_attr(drmaa_remote_command=/usr/local/packages/pulsar/etc/files/staging/50/command.sh)

t #78bc [   539.97] <- drmaa_set_attribute =0
t #78bc [   539.97] -> drmaa_set_attribute(jt=0x7f69cc002f80, 
name='drmaa_output_path', value=':/usr/local/packages/pulsar/etc/fi\

les/staging/50/stdout')
t #78bc [   539.97] -> 
fsd_template_set_attr(drmaa_output_path=:/usr/local/packages/pulsar/etc/files/staging/50/stdout)

t #78bc [   539.97] <- drmaa_set_attribute =0
t #78bc [   539.97] -> drmaa_set_attribute(jt=0x7f69cc002f80, 
name='drmaa_job_name', value='pulsar_50')

t #78bc [   539.97] -> fsd_template_set_attr(drmaa_job_name=pulsar_50)
t #78bc [   539.97] <- drmaa_set_attribute =0
t #78bc [   539.97] -> drmaa_set_attribute(jt=0x7f69cc002f80, 
name='drmaa_error_path', value=':/usr/local/packages/pulsar/etc/fil\

es/staging/50/stderr')
t #78bc [   539.97] -> 

Re: [galaxy-dev] Running unit tests on [dev]

[Marius van den Beek]

Hi Max,

that's './run_tests.sh -unit' (from the root of the clone).
You can also go directly through nose (pip install nose)
and running nosetests test/unit/

Best,
Marius

2016-11-24 17:42 GMT+01:00 Maximilian Friedersdorff :

>
>
> 
> Un o’r 4 prifysgol uchaf yn y DU a’r orau yng Nghymru am fodlonrwydd
> myfyrwyr.
> (Arolwg Cenedlaethol y Myfyrwyr 2016)
> www.aber.ac.uk
>
> Top 4 UK university and best in Wales for student satisfaction
> (National Student Survey 2016)
> www.aber.ac.uk
>
>
> -- Forwarded message --
> From: Maximilian Friedersdorff 
> To: Galaxy Dev List 
> Cc:
> Date: Thu, 24 Nov 2016 16:42:27 +
> Subject: Running unit tests on [dev]
> Hi,
>
> what is the recommended procedure for running unit tests on the current
> dev branch (or testing in general)?  The wiki suggests
> `run_unit_tests.sh`, which doesn't exist.  The file `./test/TESTING.md`
> suggests reading the wiki.
>
> Many Thanks
>
> Max
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Installing ToolShed Tools into Galaxy via Browser Interface Exception Thrown

[Marius van den Beek]

Hello Miuki,

this looks like your integrated_tool_panel.xml file has some problems.
Can you try to move this file to a different location and try the 
installation again?

It should be located in your galaxy directory.

Best,
Marius


On 11/03/2016 03:16 PM, Yip, Miu ki wrote:

Hi all,

While trying to install tools from the toolshed via the browser interface, an 
error keeps appearing and nothing is abled to be installed. Currently using the 
latest version of Galaxy and having tried to install several different tools, 
the following error keeps appearing in the log file.

galaxy.queue_worker DEBUG 2016-11-03 13:08:32,425 Executing toolbox reload on 
'main'
Exception in thread Thread-2:
Traceback (most recent call last):
   File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
 self.run()
   File "/usr/lib64/python2.7/threading.py", line 764, in run
 self.__target(*self.__args, **self.__kwargs)
   File 
"/mnt/sdd1/nondocker_galaxy/galaxy/lib/galaxy/tools/toolbox/watcher.py", line 
147, in on_any_event
 self._handle(event)
   File 
"/mnt/sdd1/nondocker_galaxy/galaxy/lib/galaxy/tools/toolbox/watcher.py", line 
150, in _handle
 self.reload_callback()
   File "/mnt/sdd1/nondocker_galaxy/galaxy/lib/galaxy/tools/toolbox/base.py", line 
78, in 
 self._tool_conf_watcher = get_tool_conf_watcher( lambda: 
reload_toolbox(app))
   File "/mnt/sdd1/nondocker_galaxy/galaxy/lib/galaxy/queue_worker.py", line 
56, in reload_toolbox
 app.toolbox = _get_new_toolbox(app)
   File "/mnt/sdd1/nondocker_galaxy/galaxy/lib/galaxy/queue_worker.py", line 
72, in _get_new_toolbox
 new_toolbox = tools.ToolBox(tool_configs, app.config.tool_path, app, 
app.toolbox._tool_conf_watcher)
   File "/mnt/sdd1/nondocker_galaxy/galaxy/lib/galaxy/tools/__init__.py", line 
110, in __init__
 tool_conf_watcher=tool_conf_watcher
   File "/mnt/sdd1/nondocker_galaxy/galaxy/lib/galaxy/tools/toolbox/base.py", 
line 1037, in __init__
 super(BaseGalaxyToolBox, self).__init__(config_filenames, tool_root_dir, 
app, tool_conf_watcher=tool_conf_watcher)
   File "/mnt/sdd1/nondocker_galaxy/galaxy/lib/galaxy/tools/toolbox/base.py", 
line 69, in __init__
 self._init_integrated_tool_panel( app.config )
   File 
"/mnt/sdd1/nondocker_galaxy/galaxy/lib/galaxy/tools/toolbox/integrated_panel.py",
 line 36, in _init_integrated_tool_panel
 self._load_integrated_tool_panel_keys()
   File "/mnt/sdd1/nondocker_galaxy/galaxy/lib/galaxy/tools/toolbox/base.py", 
line 368, in _load_integrated_tool_panel_keys
 tree = parse_xml( self._integrated_tool_panel_config )
   File "/mnt/sdd1/nondocker_galaxy/galaxy/lib/galaxy/util/__init__.py", line 
211, in parse_xml
 root = tree.parse( fname, parser=ElementTree.XMLParser( 
target=DoctypeSafeCallbackTarget() ) )
   File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 656, in parse
 parser.feed(data)
   File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1642, in feed
 self._raiseerror(v)
   File "/usr/lib64/python2.7/xml/etree/ElementTree.py", line 1506, in 
_raiseerror
 raise err
ParseError: not well-formed (invalid token): line 44, column 71

Is there an XML or configuration file which needs to be changed for this to 
work? Not sure how to proceed.

Thanks!
Miuki
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
   https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
   http://galaxyproject.org/search/mailinglists/


___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] user email info

[Marius van den Beek]

Hi Mohamed,

“$__user_email__” is filled in by galaxy when generating the command line,
so you can directly refer to this in your command line.
See 
https://github.com/galaxyproject/tools-iuc/blob/master/tools/tool_factory_2/rgToolFactory2.xml#L12
 

for an example.

Best,
Marius

> On 20 Oct 2016, at 14:55, Mohamed Kassam  wrote:
> 
> Dear all,
> I would like to send as a parameter the name of the user (either user email 
> or username).
> I tried in my xml file the option 
> 
> 
> 
>   
> 
>   
> 
> But  I have in results 
> X__user_email__
> 
> 
> Any clue ?
> 
> 
> Thanks in advance,
> 
> Mohamed
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
> 
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Changing Tool Shed URL

[Marius van den Beek]

This wouldn’t be a transfer, you would uninstall the tools coming from the
old toolshed and re-install them form the new toolshed.
This is only possible if these tools are available from the second toolshed.

To illustrate the procedure, this is what you would do if you would take
all tools from usegalaxy.org,
and replace the tools coming from the main tool shed with the same tools
but coming form the testtoolshed

get-tool-list -g https://usegalaxy.org/ -o tool_list.yml
sed -i .bak 's/toolshed.g2.bx.psu.edu/testtoolshed.g2.bx.psu.edu/g'
tool_list.yml
shed-install -g  -a  -t tool_list.yml

​

On 20 October 2016 at 11:02, Olivia Doppelt-Azeroual <
olivia.dopp...@pasteur.fr> wrote:

> Hello Marius,
>
> Thank you for your answer.
>
> Are you sure that ephemeris can transfer tool from toolshed1 to toolshed2
> ? We read that it was only implemented to install tools from a toolshed on
> a Galaxy.
>
> We will try the second solution and as we already tried to manipulate the
> database, we stongly agree that it is not THE solution.
>
> Cheers,
>
> -
>
> Fabien and Olivia and Youssef
>
>
>
> Le 20/10/2016 à 10:29, Marius van den Beek a écrit :
>
> Hi Youssef,
>
> the tool shed url is also stored in the database, that's why just changing
> the shed_tool_conf.xml is not working. I think you have a bunch of options
> here:
> 1: You can make a list of tools from that toolshed and re-install them
> from the new toolshed.
> https://pypi.python.org/pypi/ephemeris/ is a nice tool to do this
> efficiently.
> Galaxy is (or should be) smart enough to recognize this and will (should)
> not break existing workflows and tool reruns.
> 2: You can setup an alias in your /etc/hosts file if only the toolshed's
> URL has changed, or you can setup a proxy to forward the requests
> if also the subdirectory has changed.
> 3. You can try to manipulate the database directly. (But I would seriously
> advise you to go with possibilities 1 or 2).
>
> Best,
> Marius
>
> On 20 October 2016 at 00:16, Youssef GHORBAL <youssef.ghor...@pasteur.fr>
> wrote:
>
>> Hello,
>>
>> I’m wondering if there is any way to change an established Tool
>> Shed URL.
>> I have a Galaxy instance and a Tool Shed instance (
>> http://mytoolshed.url/xxx) A lot of tools have beed installed from that
>> Tool Shed inside Galaxy (tools referenced inside workflows etc)
>> I want to change the Tool Shed URL from http://mytoolshed.url/xxx
>> to http://mytoolshed2.url/.
>>
>> Galaxy and Tool Shed seems to use parts of the URL as a guid for
>> tools.
>>
>> Tool Shed does not seem to bother to be called by either URLs (it
>> generates guids with the new URLs scheam for new tools though)
>>
>> Galaxy does not seem to handle it, I changed the
>> tool_sheds_conf.xml to point to new URL, but when I update a tool on the
>> Tool Shed, Galaxy gets the new version but it generates buggy tool panel
>> (the tool is cited twice)
>> I also tried to mess around with the shed_tool_conf.xml apdapting
>> the guids and the  items with no luck.
>>
>> Can someone point me to the right direction ? How the matching is
>> done between shed_tool_conf.xml entries, the tool_version and
>> tool_shed_repository tables.
>>
>> Thank you for your help.
>>
>> Youssef Ghorbal
>> Institut Pasteur
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
>
> --
> Olivia Doppelt-Azeroual, PhD
> Bioinformatics Engineer
> CIB - C3BI - Institut Pasteur, Paris
>
> Email: olivia.dopp...@pasteur.fr
> Tel: 01 44 38 92 15
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Changing Tool Shed URL

[Marius van den Beek]

Hi Youssef,

the tool shed url is also stored in the database, that's why just changing
the shed_tool_conf.xml is not working. I think you have a bunch of options
here:
1: You can make a list of tools from that toolshed and re-install them from
the new toolshed.
https://pypi.python.org/pypi/ephemeris/ is a nice tool to do this
efficiently.
Galaxy is (or should be) smart enough to recognize this and will (should)
not break existing workflows and tool reruns.
2: You can setup an alias in your /etc/hosts file if only the toolshed's
URL has changed, or you can setup a proxy to forward the requests
if also the subdirectory has changed.
3. You can try to manipulate the database directly. (But I would seriously
advise you to go with possibilities 1 or 2).

Best,
Marius

On 20 October 2016 at 00:16, Youssef GHORBAL 
wrote:

> Hello,
>
> I’m wondering if there is any way to change an established Tool
> Shed URL.
> I have a Galaxy instance and a Tool Shed instance (
> http://mytoolshed.url/xxx) A lot of tools have beed installed from that
> Tool Shed inside Galaxy (tools referenced inside workflows etc)
> I want to change the Tool Shed URL from http://mytoolshed.url/xxx
> to http://mytoolshed2.url/.
>
> Galaxy and Tool Shed seems to use parts of the URL as a guid for
> tools.
>
> Tool Shed does not seem to bother to be called by either URLs (it
> generates guids with the new URLs scheam for new tools though)
>
> Galaxy does not seem to handle it, I changed the
> tool_sheds_conf.xml to point to new URL, but when I update a tool on the
> Tool Shed, Galaxy gets the new version but it generates buggy tool panel
> (the tool is cited twice)
> I also tried to mess around with the shed_tool_conf.xml apdapting
> the guids and the  items with no luck.
>
> Can someone point me to the right direction ? How the matching is
> done between shed_tool_conf.xml entries, the tool_version and
> tool_shed_repository tables.
>
> Thank you for your help.
>
> Youssef Ghorbal
> Institut Pasteur
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Conda: confused dependencies with multiple R versions and rpy

[Marius van den Beek]

Hi Steve,

The scatterplot tool is relatively old and has not had its dependencies
updated to be conda compatible.
https://github.com/galaxyproject/tools-devteam/blob/master/tools/scatterplot/scatterplot.xml

The relevant lines are these:

  
numpy
rpy
   wrote:

> Hi,
>   trying to get my tools going, part 29.  I have a Docker image with the
> latest Galaxy release, patched to fix the Conda install bug I found
> earlier, my tools are installed ok and I’ve added a few from the toolshed
> for good measure.
>
> One of my tools requires R 3.0.2 which is installed ok.   I also installed
> scatterplot from the toolshed which requires R 2.11; that is also installed
> ok by conda. However, I get this message when trying to run scatterplot:
>
> Traceback (most recent call last):
>   File 
> "/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/scatterplot/d243056b22ed/scatterplot/scatterplot.py",
>  line 5, in 
> from rpy import *
>   File 
> "/tool_deps/rpy/1.0.3/devteam/package_rpy_1_0_3/82170c94ca7c/lib/python/rpy.py",
>  line 134, in 
> """ % RVERSION)
> RuntimeError: No module named _rpy3002
>
>   RPy module can not be imported. Please check if your rpy
>   installation supports R 3.0.2. If you have multiple R versions
>   installed, you may need to set RHOME before importing rpy. For
>   example:
>
>   >>> from rpy_options import set_options
>   >>> set_options(RHOME='c:/progra~1/r/rw2011/')
>   >>> from rpy import *
>
>
> So it seems that rpy is installed and is aware of one version of R but not
> the other, and the other is being found (perhaps because it’s the default
> install in the Docker image?)
>
> FYI my docker image recipe is here:
>
> https://github.com/Alveo/docker-galaxy-alveo
>
> and the image itself is here:
>
> https://hub.docker.com/r/stevecassidy/alveo-galaxy/
>
> Any pointers to the way forward appreciated.
>
> Thanks,
> Steve
> —
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Conda: confused dependencies with multiple R versions and rpy

[Marius van den Beek]

Forgot to paste the link:
https://github.com/galaxyproject/tools-devteam/issues/385

On 1 September 2016 at 11:08, Marius van den Beek <m.vandenb...@gmail.com>
wrote:

> Hi Steve,
>
> The scatterplot tool is relatively old and has not had its dependencies
> updated to be conda compatible.
> https://github.com/galaxyproject/tools-devteam/
> blob/master/tools/scatterplot/scatterplot.xml
>
> The relevant lines are these:
>
>   
> numpy
> rpy
>   
>
>
> So it would attempt to get the R version via the rpy package. So my guess
> is that there is no rpy(1)
> package in conda. We can try to update the tool to use rpy2 and hope that
> that'll just work,
> and if that doesn't work we can explicitly add a requirement on R (which
> we can get through conda).
> I've opened an issue here:
>
> Best,
> Marius
>
> On 1 September 2016 at 09:50, Steve Cassidy <steve.cass...@mq.edu.au>
> wrote:
>
>> Hi,
>>   trying to get my tools going, part 29.  I have a Docker image with the
>> latest Galaxy release, patched to fix the Conda install bug I found
>> earlier, my tools are installed ok and I’ve added a few from the toolshed
>> for good measure.
>>
>> One of my tools requires R 3.0.2 which is installed ok.   I also
>> installed scatterplot from the toolshed which requires R 2.11; that is also
>> installed ok by conda. However, I get this message when trying to run
>> scatterplot:
>>
>> Traceback (most recent call last):
>>   File 
>> "/shed_tools/toolshed.g2.bx.psu.edu/repos/devteam/scatterplot/d243056b22ed/scatterplot/scatterplot.py",
>>  line 5, in 
>> from rpy import *
>>   File 
>> "/tool_deps/rpy/1.0.3/devteam/package_rpy_1_0_3/82170c94ca7c/lib/python/rpy.py",
>>  line 134, in 
>> """ % RVERSION)
>> RuntimeError: No module named _rpy3002
>>
>>   RPy module can not be imported. Please check if your rpy
>>   installation supports R 3.0.2. If you have multiple R versions
>>   installed, you may need to set RHOME before importing rpy. For
>>   example:
>>
>>   >>> from rpy_options import set_options
>>   >>> set_options(RHOME='c:/progra~1/r/rw2011/')
>>   >>> from rpy import *
>>
>>
>> So it seems that rpy is installed and is aware of one version of R but
>> not the other, and the other is being found (perhaps because it’s the
>> default install in the Docker image?)
>>
>> FYI my docker image recipe is here:
>>
>> https://github.com/Alveo/docker-galaxy-alveo
>>
>> and the image itself is here:
>>
>> https://hub.docker.com/r/stevecassidy/alveo-galaxy/
>>
>> Any pointers to the way forward appreciated.
>>
>> Thanks,
>> Steve
>> —
>> Department of Computing, Macquarie University
>> http://web.science.mq.edu.au/~cassidy
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] my.galaxy/toolshed bioblend or install_tool_shed_tools swallows toolshed

[Marius van den Beek]

I have also added the addition of a trailing slash to ephemeris. Thanks for
reporting this.

On Aug 31, 2016 7:43 PM, "Nicola Soranzo" <nsora...@tiscali.it> wrote:

> Hi Ido,
> your analysis is correct, this is what the BioBlend documentation
> prescribes:
>
> https://bioblend.readthedocs.io/en/latest/api_docs/galaxy/
> all.html#bioblend.galaxy.GalaxyInstance
>
> Cheers,
> Nicola
>
>  Tamir,Ido ha scritto 
>
> >It only works when specifying the url like this (http + trailing slash)
> otherwise urljoin removes the subdir:
> >'http://server/subdir/'
> >
> >and I use ephemeris
> >
> >thank you very much,
> >ido
> >
> >> On 30 Aug 2016, at 17:55, Marius van den Beek <m.vandenb...@gmail.com>
> wrote:
> >>
> >> Hi Ido,
> >>
> >> please use ephemeris, it’s as easy as it gets:
> >> pip install ephemeris
> >> shed-install -g  -a  -t .
> >> The install_tool_shed_tools.py is deprecated and will not work anymore.
> >> That said, we haven’t tested installing from toolshed on
> subdirectories, it’s entirely possible that there is a problem.
> >> I will have a look and let you know.
> >>
> >> Best,
> >> Marius
> >>
> >>
> >> On 30 August 2016 at 17:14, Tamir,Ido <ido.ta...@vbcf.ac.at> wrote:
> >> Hi,
> >>
> >> a)
> >> short version:
> >> instead of:
> >> http://gecko.imp.univie.ac.at/galtools/api/repositories/get_
> ordered_installable_revisions?owner=itme=shrnaseqw
> >>
> >> the tool makes a request to:
> >> http://gecko.imp.univie.ac.at/api/repositories/get_ordered_
> installable_revisions?owner=itme=shrnaseqw
> >>
> >> (i.e. galtools is missing)
> >>
> >>
> >>
> >>
> >> b)
> >> long version:
> >>
> >> my toolshed is running behind a proxy under:
> >> my.server/galtools
> >>
> >> I use the script install_tool_shed_tools.py via galaxy-ansible-playbook
> (I know of ephemeris but don’t know how to use it yet).
> >>
> >> Installing tools from the default toolshed works, but when I try to
> access my own tools I see in nxginx logs
> >> that the request from the script swallowed the “galtools” part.
> >>
> >> But the url is stored correctly in json, and I can get the appropriate
> response when I specify the url in a browser:
> >>
> >> http://gecko.imp.univie.ac.at/galtools/api/repositories/get_
> ordered_installable_revisions?owner=itme=shrnaseqw
> >>
> >> =>
> >>
> >> ["f663fded0c69", "4aeaa49154f8"]
> >>
> >>
> >> thank you very much for your input,
> >> ido
> >>
> >>
> >> ansible log:
> >> ok: [localhost] => (item={u'owner': u'itme', u'tool_shed_url': u'
> gecko.imp.univie.ac.at/galtools', u'tool_panel_section_id': u'ngs_rna',
> u'name': u'shrnaseqw'}) => {"changed": false, "cmd":
> ["/groups/vbcf-ngs/wfsys/galaxy/galaxy/.venv/bin/python",
> "install_tool_shed_tools.py", "-y", "name: shrnaseqw\nowner:
> itme\ntool_panel_section_id: ngs_rna\ntool_shed_url:
> gecko.imp.univie.ac.at/galtools\n", "-a", "7e7c8ec27a0a99acee883d2beea8a60b",
> "-g", "http://127.0.0.1:9098/;], "delta": "0:00:00.820941", "end":
> "2016-08-30 16:55:58.869977", "failed": false, "failed_when_result": false,
> "invocation": {"module_args": {"_raw_params": 
> "/groups/vbcf-ngs/wfsys/galaxy/galaxy/.venv/bin/python
> install_tool_shed_tools.py -y \"name: shrnaseqw\nowner:
> itme\ntool_panel_section_id: ngs_rna\ntool_shed_url:
> gecko.imp.univie.ac.at/galtools\n\" -a 7e7c8ec27a0a99acee883d2beea8a60b
> -g http://127.0.0.1:9098/;, "_uses_shell": false, "chdir":
> "/groups/vbcf-ngs/wfsys/galaxy/galaxy", "creates": null, "executable":
> null, "removes": null, "warn": true}, "module_name": "command"}, "item":
> {"name": "shrnaseqw", "owner": "itme", "tool_panel_section_id": "ngs_rna",
> "tool_shed_url": "gecko.imp.univie.ac.at/galtools"}, "rc": 1, "start":
> "2016-08-30 16:55:58.049036", "stderr": "Traceback (most recent call
> last):\n  File \"install_tool_shed_tools.py\", line 652, in \n
> 

Re: [galaxy-dev] Putting it all together - toolshed tool + conda dependency

[Marius van den Beek]

Hi Steve,

Looks like you're running an older version of galaxy in the docker
container, since newer galaxy will build the metadata environment in a
separate environment called 'conda-metadata-env',

and we have also changed the way that the handlers receive their Python
environment (that's why sqlalachemy couldn't be found).

Can you try updating the container or building from the dev branch? I think
the necessary changes for conda should be in the dev branch, which you can
pull with docker pull quay.io/bgruening/galaxy:dev.

Best,

Marius

On Aug 26, 2016 6:49 AM, "Steve Cassidy"  wrote:

> Ok, probing further trying to understand this. It looks like the job
> runner is trying to call set_meta() from galaxy_ext/metadata/set_metadata.py.
> In there there’s a line that tries to set up the PATH:
>
> # insert *this* galaxy before all others on sys.path
> sys.path.insert( 1, os.path.abspath( os.path.join( os.path.dirname(
> __file__ ), os.pardir, os.pardir ) ) )
>
> In my case the result is:
>
> ['/export/galaxy-central/database/job_working_directory/000/3’,
> '/galaxy-central/lib’,
> '/galaxy-central/lib’,
>  '/export/galaxy-central/database/job_working_directory/000/
> 3/conda-env/lib/python27.zip’,
>  '/export/galaxy-central/database/job_working_directory/000/
> 3/conda-env/lib/python2.7’,
>  '/export/galaxy-central/database/job_working_directory/000/
> 3/conda-env/lib/python2.7/plat-linux2’,
>  '/export/galaxy-central/database/job_working_directory/000/
> 3/conda-env/lib/python2.7/lib-tk’,
> '/export/galaxy-central/database/job_working_directory/000/3
> /conda-env/lib/python2.7/lib-old’,
>  '/export/galaxy-central/database/job_working_directory/000/
> 3/conda-env/lib/python2.7/lib-dynload’,
> '/export/galaxy-central/database/job_working_directory/000/3
> /conda-env/lib/python2.7/site-packages’,
> '/export/galaxy-central/database/job_working_directory/000/3
> /conda-env/lib/python2.7/site-packages/setuptools-25.1.6-py2.7.egg’
> ]
>
> Looking for sqlalchemy, I find it installed at:
>
> /galaxy-central/.venv/lib/python2.7/site-packages/sqlalchemy
>
> so clearly that line of code is not doing what it should. I’m guessing
> that the Conda resolver is running the script within a conda env and the
> BETA magic hasn’t quite made it to the right place yet…
>
> BTW I was trying to understand the flags you mentioned but I can’t find
> GALAXY_CONFIG_ENABLE_BETA_TOOL_COMMAND_ISOLATION anywhere other than in
> sample Dockerfiles - is this some kind of magic incantation that creates a
> rift in space-time…enquiring minds want to know!!!
>
> Steve
> —
> Department of Computing, Macquarie University
> http://web.science.mq.edu.au/~cassidy
>
> > On 25 Aug 2016, at 9:54 PM, Steve Cassidy 
> wrote:
> >
> > Nah, definitely baby steps…
> >
> > so, in the repo you point to there seems to be an error in the
> Dockerfile, the ENV line should use the var=value syntax to have more than
> one setting on one line (maybe that’s changed recently in docker).
> >
> > with this I built a new docker image, when I run my first tool it takes
> an age while it’s installing the deps, then it crashes with:
> >
> > Traceback (most recent call last):
> >  File 
> > "/export/galaxy-central/database/job_working_directory/000/1/set_metadata_QUJQLD.py",
> line 1, in 
> >from galaxy_ext.metadata.set_metadata import set_metadata;
> set_metadata()
> >  File "/galaxy-central/lib/galaxy_ext/metadata/set_metadata.py", line
> 23, in 
> >from sqlalchemy.orm import clear_mappers
> > ImportError: No module named sqlalchemy.orm
> >
> > and the output:
> >
> > discarding /galaxy-central/tool_deps/_conda/bin from PATH
> > prepending 
> > /export/galaxy-central/database/job_working_directory/000/1/conda-env/bin
> to PATH
> > discarding /galaxy-central/tool_deps/_conda/bin from PATH
> > prepending 
> > /export/galaxy-central/database/job_working_directory/000/1/conda-env/bin
> to PATH
> >
> > I’m guessing this is some kind of conflict between python versions in
> and out of conda environments? Surely sqlalchemy would be installed for
> Galaxy to work?
> >
> > I’ll try to dig around this in the morning but if it rings a bell…
> >
> > Steve
> > —
> > Department of Computing, Macquarie University
> > http://web.science.mq.edu.au/~cassidy
> >
> >> On 25 Aug 2016, at 8:43 PM, Björn Grüning 
> wrote:
> >>
> >> Hi Steve,
> >>
> >> you call this baby-steps? I think this is huge! :)
> >>
> >> All what you are missing is to enable conda in Galaxy.
> >> Look at Gregs new flavour which is entirely Conda/Galaxy based.
> >>
> >> You need to enable these env vars to make Galaxy conda enabled:
> >>
> >> https://github.com/gregvonkuster/docker-galaxy-csg/blob/mast
> er/Dockerfile#L9
> >>
> >> Hope this helps,
> >> Bjoern
> >>
> >> Am 25.08.2016 um 12:32 schrieb Steve Cassidy:
> >>> Hi all,
> >>> I’m making baby steps towards having a repeatable installation for my
> >>> tools.  But I’m now stuck, 

Re: [galaxy-dev] Python tool wrapper with multiple input and output files

[Marius van den Beek]

Hi Marco,

The tag has been deprecated,
what you should use is just

  

  

Note the $*tool_directory*, this is required for finding the path to the
script.
(And that has been “magically” done for you by the tag).

Hope that works for you,
Marius

On 8 July 2016 at 11:43, Marco Tangaro <ma.tang...@gmail.com> wrote:

> Dear Marius and all,
>
> thanks a lot for your answer, it is indeed very interesting.
>
> I'm following this one
>
> https://github.com/galaxyproject/galaxy/blob/dev/test/functional/tools/collection_two_paired.xml
> to create an output list, starting from an input list.
>
>
> The command section of my script_wrapper.xml looks like this:
>
>   
> 
>   
>
>
> Once I've run the tool in galaxy the job command line looks like:
>
> python /home/galaxy/galaxy/tools/script-collection/script_wrapper.py -i
> /home/galaxy/galaxy/database/files/000/dataset_269.dat -f
> "/home/galaxy/galaxy/database/files/000/dataset_256.dat" -1
> "/home/galaxy/galaxy/database/files/000/dataset_312.dat" -t
> "${GALAXY_SLOTS:-4}"; script_wrapper.py -i
> /home/galaxy/galaxy/database/files/000/dataset_254.dat -f
> "/home/galaxy/galaxy/database/files/000/dataset_256.dat" -1
> "/home/galaxy/galaxy/database/files/000/dataset_313.dat" -t
> "${GALAXY_SLOTS:-4}";
>
> And the output looks fine only for the first command in the job comman
> line, becasue "python" is called only at the first iteration of the for
> cycle.
>
> Is there a way to prevent this behaviour or some "best practices" ?
>
> Thanks a lot,
> Marco.
>
> 2016-06-07 14:05 GMT+02:00 Marius van den Beek <m.vandenb...@gmail.com>:
>
>> Hi Marco,
>>
>> you've got an interesting use-case there.
>> You may want to use either a dataset list (if you only supply rna_n.bam),
>> or a paired dataset list (rna_n.bam and dna_n.bam).
>> I would probably implement a conditional, where the user selects either a
>> dataset list or a paired dataset list.
>> The output would then be another collection of output files.
>> Have a look at the test tool folder, and see if any of the tools named
>> collection_*.xml fits what you would like to do
>> https://github.com/galaxyproject/galaxy/tree/dev/test/functional/tools
>> These two may be a good basis for what you want to achieve:
>>
>> https://github.com/galaxyproject/galaxy/blob/dev/test/functional/tools/collection_creates_list.xml
>> [this one creates an output collection]
>>
>> https://github.com/galaxyproject/galaxy/blob/dev/test/functional/tools/collection_two_paired.xml
>> [this one has a conditional to either select a list or a paired list as
>> input]
>>
>> Let us know if you need more help!
>>
>> Cheers,
>> Marius
>>
>> On 7 June 2016 at 09:50, Marco Tangaro <ma.tang...@gmail.com> wrote:
>>
>>> Dear experts,
>>> my name is Marco and I'm working to port our python tool to the Galaxy
>>> framework.
>>> The main script needs a rna.bam file as input, a reference fasta file,
>>> both mandatory. Finally, you can add a dna.bam file, but this is optional.
>>> Therefore an example command is:
>>>
>>> script.py -i rna.bam -f reference.fa -j dna.bam
>>>
>>> The outout is a tabular.
>>> Again the -j dna.bam option is completely optional.
>>> So quite soon it turned out that I had to use a python wrapper to parse
>>> our script. Now the wrapper works fine.
>>>
>>>
>>> The next step is to run the tool over multiple input file and we would
>>> like to avoid to use a workflow.
>>>
>>> The idea is that to each input file corresponds an output file. The
>>> reference is still the same.
>>> For instance, we have:
>>>
>>> rna_1.bam + dna_1.bam -> output_1.txt
>>> rna_2.bam + dna_2.bam -> output_2.txt
>>> rna_3.bam + dna_3.bam -> output_3.txt
>>> ...
>>> and so on.
>>>
>>>
>>> But I don't know the best strategy to give to my wrapper multiple input
>>> files.
>>> Moreover I have to be sure, when the dna_xyz.bam files are uploaded,
>>> that they correspond to the right rna_xyz.bam file.
>>>
>>> I would like to have as output a page which is showing as results the
>>> link to the single output files as suggested here.
>>> https://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files
>>> planning to integrate a javascript interface.
>>>
>>> I've browsed a lot, but on multiple input file the posts are 

Re: [galaxy-dev] Python tool wrapper with multiple input and output files

[Marius van den Beek]

Hi Marco,

you've got an interesting use-case there.
You may want to use either a dataset list (if you only supply rna_n.bam),
or a paired dataset list (rna_n.bam and dna_n.bam).
I would probably implement a conditional, where the user selects either a
dataset list or a paired dataset list.
The output would then be another collection of output files.
Have a look at the test tool folder, and see if any of the tools named
collection_*.xml fits what you would like to do
https://github.com/galaxyproject/galaxy/tree/dev/test/functional/tools
These two may be a good basis for what you want to achieve:
https://github.com/galaxyproject/galaxy/blob/dev/test/functional/tools/collection_creates_list.xml
[this one creates an output collection]
https://github.com/galaxyproject/galaxy/blob/dev/test/functional/tools/collection_two_paired.xml
[this one has a conditional to either select a list or a paired list as
input]

Let us know if you need more help!

Cheers,
Marius

On 7 June 2016 at 09:50, Marco Tangaro  wrote:

> Dear experts,
> my name is Marco and I'm working to port our python tool to the Galaxy
> framework.
> The main script needs a rna.bam file as input, a reference fasta file,
> both mandatory. Finally, you can add a dna.bam file, but this is optional.
> Therefore an example command is:
>
> script.py -i rna.bam -f reference.fa -j dna.bam
>
> The outout is a tabular.
> Again the -j dna.bam option is completely optional.
> So quite soon it turned out that I had to use a python wrapper to parse
> our script. Now the wrapper works fine.
>
>
> The next step is to run the tool over multiple input file and we would
> like to avoid to use a workflow.
>
> The idea is that to each input file corresponds an output file. The
> reference is still the same.
> For instance, we have:
>
> rna_1.bam + dna_1.bam -> output_1.txt
> rna_2.bam + dna_2.bam -> output_2.txt
> rna_3.bam + dna_3.bam -> output_3.txt
> ...
> and so on.
>
>
> But I don't know the best strategy to give to my wrapper multiple input
> files.
> Moreover I have to be sure, when the dna_xyz.bam files are uploaded, that
> they correspond to the right rna_xyz.bam file.
>
> I would like to have as output a page which is showing as results the link
> to the single output files as suggested here.
> https://wiki.galaxyproject.org/Admin/Tools/Multiple%20Output%20Files
> planning to integrate a javascript interface.
>
> I've browsed a lot, but on multiple input file the posts are old.
> I'm using the last galaxy release (16_04).
>
> I'm quite new to the galaxy world...
> Thanks a lot for your suggestions,
> Marco
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] not sure what I can do about this...

[Marius van den Beek]

Hi Edgar,

it’s a good idea to use python2.7 for galaxy, since 16.01 is the last
release supporting python 2.6.
In general you can create a new virrtualenv with a a specific python
interpreter like this:

virtualenv -p  

If you create a new new virtualenv .venv in your galaxy source folder,
run.sh will pick this up.
If you are using a different way to start galaxy, you can use something
along the lines of

/scripts/paster.py
serve config/galaxy.ini>

Hope that will get you past the problems starting up galaxy.

Cheers,
Marius
​

On 19 May 2016 at 14:48, Fernandez Edgar 
wrote:

> Hey Guys,
>
>
>
> I’ve fixed my first problem with the environment.
>
> But I’m still getting the second one.
>
>
>
> Also, did any of you guys try to deploy galaxy with Microsoft Azure?
>
> I’m actually looking into it and I was wondering if you had any input…
>
>
>
> Let me know if you have any info on my libchtslib.so: input error or
> Microsoft Azure.
>
>
>
> Cheers!
>
>
>
> *Edgar Fernandez*
>
> System Administrator (Linux)
>
> Direction Générale des Technologies de l'Information et de la Communication
>
> *Université de Montréal*
>
>
>
> PAVILLON ROGER-GAUDRY, bureau X-210
>
> (  Bur. : *1-514-343-6111 poste 16568*
>
>
>
>
>
> *From:* Fernandez Edgar
> *Sent:* May-18-16 2:00 PM
> *To:* galaxy-...@bx.psu.edu
> *Subject:* RE: [galaxy-dev] not sure what I can do about this...
>
>
>
> Hey Guys,
>
>
>
> Hope everyone is doing well.
>
>
>
> So I’m back at galaxy v16.01
>
>
>
> I’m having a small problem.
>
> My python system version is 2.6.6
>
>
>
> So I installed python version 2.7.8 from  Software Collection Library
> release 1.2 packages for Oracle Linux 6 (x86_64)
> 
>
> This means I need to source an environment before being about to use it.
>
> So what I’ve created a file config/local_env.sh with:
>
> *# Source python27*
>
> *export PATH=/opt/rh/python27/root/usr/bin${PATH:+:${PATH}}*
>
> *export
> LD_LIBRARY_PATH=/opt/rh/python27/root/usr/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}*
>
> *export MANPATH=/opt/rh/python27/root/usr/share/man:${MANPATH}*
>
> *# For systemtap*
>
> *export
> XDG_DATA_DIRS=/opt/rh/python27/root/usr/share${XDG_DATA_DIRS:+:${XDG_DATA_DIRS}}*
>
> *# For pkg-config*
>
> *export
> PKG_CONFIG_PATH=/opt/rh/python27/root/usr/lib64/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}*
>
>
>
> These export comes directly with the installation of python 2.7.8 in the
> file /opt/rh/python27/enable.
>
> But when I started my galaxy it didn’t stick.
>
>
>
> I had to insert those exports into the file run.sh right before the
> following line:
>
> *./scripts/common_startup.sh $common_startup_args || exit 1*
>
>
>
> Any suggestions to make it work?
>
>
>
> Also, I get this error :
>
> *Traceback (most recent call last):*
>
> *  File "./scripts/paster.py", line 27, in *
>
> *serve.run()*
>
> *  File "/home/galaxy/galaxy-v16.01/lib/galaxy/util/pastescript/serve.py",
> line 1061, in run*
>
> *invoke(command, command_name, options, args[1:])*
>
> *  File "/home/galaxy/galaxy-v16.01/lib/galaxy/util/pastescript/serve.py",
> line 1067, in invoke*
>
> *exit_code = runner.run(args)*
>
> *  File "/home/galaxy/galaxy-v16.01/lib/galaxy/util/pastescript/serve.py",
> line 223, in run*
>
> *result = self.command()*
>
> *  File "/home/galaxy/galaxy-v16.01/lib/galaxy/util/pastescript/serve.py",
> line 639, in command*
>
> *app = loadapp( app_spec, name=app_name, relative_to=base,
> global_conf=vars)*
>
> *  File
> "/home/galaxy/galaxy-v16.01/lib/galaxy/util/pastescript/loadwsgi.py", line
> 295, in loadapp*
>
> *return loadobj(APP, uri, name=name, **kw)*
>
> *  File
> "/home/galaxy/galaxy-v16.01/lib/galaxy/util/pastescript/loadwsgi.py", line
> 319, in loadobj*
>
> *global_conf=global_conf)*
>
> *  File
> "/home/galaxy/galaxy-v16.01/lib/galaxy/util/pastescript/loadwsgi.py", line
> 344, in loadcontext*
>
> *global_conf=global_conf)*
>
> *  File
> "/home/galaxy/galaxy-v16.01/lib/galaxy/util/pastescript/loadwsgi.py", line
> 368, in _loadconfig*
>
> *return loader.get_context(object_type, name, global_conf)*
>
> *  File
> "/home/galaxy/galaxy-v16.01/lib/galaxy/util/pastescript/loadwsgi.py", line
> 506, in get_context*
>
> *section)*
>
> *  File
> "/home/galaxy/galaxy-v16.01/lib/galaxy/util/pastescript/loadwsgi.py", line
> 565, in _context_from_explicit*
>
> *value = import_string(found_expr)*
>
> *  File
> "/home/galaxy/galaxy-v16.01/lib/galaxy/util/pastescript/loadwsgi.py", line
> 125, in import_string*
>
> *return pkg_resources.EntryPoint.parse("x=" + s).load(False)*
>
> *  File
> "/home/galaxy/galaxy-v16.01/.venv/lib/python2.7/site-packages/pkg_resources.py",
> line 2260, in load*
>
> *entry = __import__(self.module_name, globals(),globals(),
> ['__name__'])*
>
> *  File "/home/galaxy/galaxy-v16.01/lib/galaxy/web/buildapp.py", line 5,
> in *
>
> *from galaxy.webapps.galaxy.buildapp 

[galaxy-dev] Fwd: Error when uploading files (using Pulsar)

[Marius van den Beek]

Ah, and if you want to figure out the ID of a tool, you can click the small
info button
of a dataset produced by the tool, so in the case of the upload tool it
reads:
Galaxy Tool ID: upload1

On 9 May 2016 at 12:25, Marius van den Beek <m.vandenb...@gmail.com> wrote:

> Puhh, you're right, and somewhere in the undepths of the wiki (I think)
> this is documented,
> but admittedly I didn't find it when searching for it.
> If you want you could add a sample of your config to
> https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced
> ,
> I believe this is something that the galaxy devteam carefully keeps up to
> date.
> You would definitely have my +1 for a Pull Request. If you don't have the
> time I can also do this for you.
>
> Marius
>
> On 9 May 2016 at 12:02, Tiziano Flati <tiziano.fl...@gmail.com> wrote:
>
>> Thank you very much, Marius, that worked!
>>
>> But how should I have known about the exact ID of the upload tool (i.e.,
>> upload1)? I think this is an extremely common pattern (upload data and
>> execute jobs on a cluster)
>>
>> 2016-05-09 11:38 GMT+02:00 Marius van den Beek <m.vandenb...@gmail.com>:
>>
>>> Hi Tiziano,
>>>
>>> I think you are correct in your assumption.
>>> If you do not need to have uploads run on pulsar, you should be able to
>>> specify a local destination for uploads (the upload tool id is upload1)
>>> in your job_conf.xml.
>>> There are some examples described here:
>>>
>>> https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced#L496
>>> If you can place a copy of galaxy and a virtualenv on your pulsar
>>> server, you could also set this up as in
>>>
>>> https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced#L379
>>>
>>> Note that I haven't tried this yet, but I think this is a good start.
>>> Let us know if that works.
>>>
>>> Cheers,
>>> Marius
>>>
>>>
>>> On 9 May 2016 at 10:42, Tiziano Flati <tiziano.fl...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I have succesfully setup a Galaxy-Pulsar architecture and I am able to
>>>> run jobs over datasets* already uploaded on a history*.
>>>>
>>>> When I try to upload a new file to the history, though, the upload job
>>>> fails with the following error:
>>>>
>>>> Traceback (most recent call last):
>>>>>   File "/home/flati/pulsar/files/staging/80/tool_files/upload.py",
>>>>> line 18, in 
>>>>> import galaxy.model # noqa
>>>>>   ImportError: No module named model
>>>>
>>>>
>>>> Note: in job_conf.xml, Pulsar is the default destination:
>>>>
>>>> 
>>>>
>>>>
>>>> Does someone know what the problem is?
>>>> I suspect that setting pulsar as the default destination causes the
>>>> upload tool to run on Pulsar's side which, however, does not have access to
>>>> Galaxy's lib directory (which contains the galaxy model module).
>>>>
>>>> Any help is very appreciated,
>>>> Tiziano
>>>>
>>>> ___
>>>> Please keep all replies on the list by using "reply all"
>>>> in your mail client.  To manage your subscriptions to this
>>>> and other Galaxy lists, please use the interface at:
>>>>   https://lists.galaxyproject.org/
>>>>
>>>> To search Galaxy mailing lists use the unified search at:
>>>>   http://galaxyproject.org/search/mailinglists/
>>>>
>>>
>>>
>>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Error when uploading files (using Pulsar)

[Marius van den Beek]

Hi Tiziano,

I think you are correct in your assumption.
If you do not need to have uploads run on pulsar, you should be able to
specify a local destination for uploads (the upload tool id is upload1) in
your job_conf.xml.
There are some examples described here:
https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced#L496
If you can place a copy of galaxy and a virtualenv on your pulsar server,
you could also set this up as in
https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced#L379

Note that I haven't tried this yet, but I think this is a good start. Let
us know if that works.

Cheers,
Marius

On 9 May 2016 at 10:42, Tiziano Flati  wrote:

> Hi all,
>
> I have succesfully setup a Galaxy-Pulsar architecture and I am able to run
> jobs over datasets* already uploaded on a history*.
>
> When I try to upload a new file to the history, though, the upload job
> fails with the following error:
>
> Traceback (most recent call last):
>>   File "/home/flati/pulsar/files/staging/80/tool_files/upload.py", line
>> 18, in 
>> import galaxy.model # noqa
>>   ImportError: No module named model
>
>
> Note: in job_conf.xml, Pulsar is the default destination:
>
> 
>
>
> Does someone know what the problem is?
> I suspect that setting pulsar as the default destination causes the upload
> tool to run on Pulsar's side which, however, does not have access to
> Galaxy's lib directory (which contains the galaxy model module).
>
> Any help is very appreciated,
> Tiziano
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] ERROR: PIP is not a legl parament of an Ansible Play

[Marius van den Beek]

Okay, that was a bit too short for the tool_list.yml.
What is important is that you either have the tool_panel_section_label or
tool_panel_section_id included in your tool_list.yml (for each tool!).
Then it would help us to see what is going on if you can run the playbook
with the - parameter, as well as what is going on at the same time in
your galaxy logs.

On 21 April 2016 at 19:30, William Ge <w...@centrilliontech.com> wrote:

> Thanks, Marius!
>
>
>
> Sure, here it is:
>
> I followed the online instructions to git clone the master Galaxy (Feb.,
> 2016) and configured it for local production use with PostgreSQL and
> proftp. First tried ansible to install the tools, but did not succeed. Then
> tried the online admin installation of tools one-by-one. That worked, and I
> installed at least one tool for each section this way. With more help from
> you and others later, I got ansible-playbook to run to install the tools.
> That ran OK, but did not see the tools installed show up on the web
> frontend.
>
>
>
> Beginning of the tool_list.yml is the same as the one included in the
> playbook/roles, except for the api_key and galaxy_instance:
>
>
>
> api_key: ‘d22b682…’
>
> galaxy_instance: 127.0.0.1:8080
>
> tools:
>
> -  name: ‘column_maker’
>
> owner: …
>
>
>
>
>
> The command I used is: ansible-playbook tools.yaml
>
> The playbook was downloaded end of March, following the link given on the
> Galaxy tool installation guide.
>
> Manual installation of the tools one-by-one from the Admin web page works.
> The tools installed show up on the web page.
>
>
>
> Thanks,
>
> Bill
>
> *From:* Marius van den Beek [mailto:m.vandenb...@gmail.com]
> *Sent:* Thursday, April 21, 2016 9:45 AM
> *To:* William Ge <w...@centrilliontech.com>
> *Cc:* Enis Afgan <enis.af...@irb.hr>; galaxy-dev@lists.galaxyproject.org
>
> *Subject:* Re: [galaxy-dev] ERROR: PIP is not a legl parament of an
> Ansible Play
>
>
>
> Hi Bill, can you describe what exactly you have done?
>
> What would be good is if you could post the beginning of your
> tool_list.yml file,
>
> the command that you used to run the playbook, the version of galaxy and
> the last commit
>
> of the playbook you are using (the last entry of `git log`).
>
> For any of this to work you should also be able manually install tools
> from the toolshed.
>
> Does this work for you?
>
>
>
> On 21 April 2016 at 18:14, William Ge <w...@centrilliontech.com> wrote:
>
> Does anyone know why Ansible installation of the tools did not update the
> shed_tools_conf.xml file to reflect the installed tools? How can I make the
> tools installed with the Ansible playbook visible on the web front end?
>
> Thanks,
>
> Bill
>
>
>
> *From:* Enis Afgan [mailto:enis.af...@irb.hr]
> *Sent:* Friday, April 8, 2016 9:19 AM
>
>
> *To:* William Ge <w...@centrilliontech.com>
> *Cc:* Marius van den Beek <m.vandenb...@gmail.com>;
> galaxy-dev@lists.galaxyproject.org
> *Subject:* Re: [galaxy-dev] ERROR: PIP is not a legl parament of an
> Ansible Play
>
>
>
> The location of that dir is defined in shed_tool_conf.xml[.sample] and
> defaults to ../shed_tools. The reasoning is described here, under item #2:
> https://wiki.galaxyproject.org/DevNewsBriefs/2012_10_23#Tool_Shed
>
>
>
> On Thu, Apr 7, 2016 at 11:09 AM, William Ge <w...@centrilliontech.com>
> wrote:
>
> Thanks, Enis!
>
> Yes, that line was uncommented with the option pointing to
> “tool_conf.xml,shed_tool_conf.xml”. However, I noticed that the
> “shed_tool_conf.xml” has not been changed after running the
> ansible-playbook to install tools. One other thing: the shed_tools
> directory was one level above the Galaxy subdirectories: i.e., all
> subdirectories (config, database, tool-data, tools, static, etc) are in
> /home/galaxy/galaxy, while “shed_tools” is in /home/galaxy. Do not know why.
>
>
>
> Thanks,
>
> Bill
>
>
>
> *From:* Enis Afgan [mailto:enis.af...@irb.hr]
> *Sent:* Thursday, April 7, 2016 7:06 AM
> *To:* William Ge <w...@centrilliontech.com>
> *Cc:* Marius van den Beek <m.vandenb...@gmail.com>;
> galaxy-dev@lists.galaxyproject.org
>
>
> *Subject:* Re: [galaxy-dev] ERROR: PIP is not a legl parament of an
> Ansible Play
>
>
>
> Hi Bill,
>
> Do you have the value set in Galaxy's *$GALAXY_ROOT/config/galaxy.ini*
> file for setting *tool_config_file *option? I believe this variable
> should be set to a file other than the default **.sample*.
>
>
>
> On Wed, Apr 6, 2016 at 9:21 PM, William Ge <w...@centrilliontech.com>
> wrote:
>
&g

Re: [galaxy-dev] init scripts version 16.01

[Marius van den Beek]

Hi Eric,

Take a look at this
https://docs.galaxyproject.org/en/master/admin/framework_dependencies.html#wheel-interaction-with-other-software
 .
I hope that should be sufficient.

Cheers,
Marius

On 11 April 2016 at 14:25, Eric Kuyt <eric.ku...@wur.nl> wrote:

> Hi Marius,
>
> Now I ran in to some problems.
>
> When trying to change metadata of a file, the python path changes (these
> tools run using drmaa) to the normal system version. Here tools are ending
> in error becaus sqlalchemy.orm cannot be found. I can ofcourse install
> sqlarchemy-python systemwide. But this woudn't make me very happy.
>
> in File: /lib/galaxy_ext/metadata/set_metadata.py
> a sys.path.insert to .venv/lib should be added. I don't know what the
> right plcae for this is.
>
> Do you hava any better ideas for this matter?
>
> Eric
>
>
>
>
> On 8 April 2016 at 16:15, Eric Kuyt <eric.ku...@wur.nl> wrote:
>
>> ​Thanks, works like a charm!​ Clean easy fix, Now why I couldn't come up
>> with this myself.
>>
>> On 2 April 2016 at 11:49, Marius van den Beek <m.vandenb...@gmail.com>
>> wrote:
>>
>>> ld have been mentioned with the release notes.
>>> I didn't test this, but it might be sufficient to replace
>>> PYTHON="/usr/bin/python"
>>> with PYTHON="".
>>> If that is sufficient for you, let us know and we will update the wiki
>>> and adjust the script in
>>>
>>
>>
>>
>>
>> --
>> Central Veterinary Institute of Wageningen UR (CVI)
>> Department of Infection Biology
>> PO box 65, 8200 AB Lelystad, NL
>> Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
>>
>> Tel:  +31-(0)320-293391
>> Fax: +31-(0)320-238153
>> E-mail: eric.ku...@wur.nl
>> Web: http://www.cvi.wur.nl
>>
>
>
>
> --
> Central Veterinary Institute of Wageningen UR (CVI)
> Department of Infection Biology
> PO box 65, 8200 AB Lelystad, NL
> Visiting address: ASG, Edelhertweg 15, 8219 PH Lelystad
>
> Tel:  +31-(0)320-293391
> Fax: +31-(0)320-238153
> E-mail: eric.ku...@wur.nl
> Web: http://www.cvi.wur.nl
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] perl version 5.18.1 repository error

[Marius van den Beek]

Hi Philipp,

I believe you have run into a rather complicated problem with perl's
dependencies.
We (the IUC) have changed from coreutils-8.22 to coreutils-8.25 for exactly
the reason you are reporting
(`...generated no output for the defined timeout period of 3600.0 seconds`).
So you should update your installation of package_perl_5_18_1.

Unfortunately, no stable release of galaxy up till now can handle this
change of dependencies cleanly.
(16.04 will be the first release that should have no problem with this).
When you are doing an update of package_perl_5_18_1, you should have
gnu-coreutils-8.25 listed as
one of perl's dependencies, as well as gnu-coreutils-8.22. For the moment
the best thing you can do is to ignore
the status of gnu-coreutils-8.22. Once you upgrade galaxy to release_16.04
or newer, repairing package_perl_5_18_1
will remove gnu-coreutils-8.22 from the list of dependencies of perl.
This should not affect the functionality of perl or meme.
Let us know how it goes

Marius

On 24 March 2016 at 09:23, Rathert, Philipp, Dr. <
philipp.rath...@ibc.uni-stuttgart.de> wrote:

> Dear Nicola,
>
>
> now it looks like package_gnu_coreutils_8_22
>
> is generating an error
>
>
>
> gnu_coreutils
> 
> 8.22 package Error
> Shutting down process id 23231 because it generated no output for the defined 
> timeout period of 3600.0 seconds.
>
> 
>
>
>
> I didn't find anything an this error on the web and I am not sure how I
> can fix this...
>
>
> cheers,
>
>
> philipp
>
> -Original message-
> *From:* Nicola Soranzo 
> *Sent:* Wednesday 23rd March 2016 12:49
> *To:* Rathert, Philipp, Dr. ;
> galaxy-dev@lists.galaxyproject.org
> *Subject:* Re: [galaxy-dev] perl version 5.18.1 repository error
>
> Hi Philipp,
> can you try to reinstall the tool dependency? Go to the Admin interface,
> "Manage installed tools", click on the
> "package_perl_xml_parser_expat_2_41", then from the "Repository Actions"
> menu select "Manage tool dependencies", select the dependency in error and
> "Uninstall", then select it again and "Install" it.
>
> Cheers,
> Nicola
>
> On 23/03/16 08:06, Rathert, Philipp, Dr. wrote:
>
> Dear all,
>
>
> i am trying to install  tool (meme-fimo) which needs 
> package_perl_xml_parser_expat_2_41
> that has a pearl 5.18.1 dependency.
>
>
> however, although the tool installs just fine it shows that 
> package_perl_xml_parser_expat_2_41
> has missing dependencies:
>
>
> Tool shed repository:package_perl_xml_parser_expat_2_41
> Tool shed repository changeset revision:1ce3818a8eb9
> Tool dependency status:Error
> Tool dependency installation error:Error installing tool dependency perl
> version 5.18.1: Unable to locate the repository directory for revision
> d905bb415eca of installed repository package_perl_5_18 owned by iuc.
> Tool dependency installation directory:
> ././tool_deps/perl/5.18.1/iuc/package_perl_xml_parser_expat_2_41/1ce3818a8eb9
>
> Is this an error of my local instance or is the revision missing on the
> toolshed?
>
> Thank you very much for your help.
>
> Cheers,
>
> philipp
>
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Trackster in 16.01 broken

[Marius van den Beek]

Hi Lubos,

I opened a Pull Request on https://github.com/galaxyproject/galaxy/pull/1930,
which should hopefully fix the problem.
If you want to give it a try, you can do
`git fetch origin pull/1930/head:more_bytestring_for_pysam`
and
`git checkout more_bytestring_for_pysam`.
When the problem has been solved, you can switch back to release_16.01 by
doing `git checkout release_16.01`.

Thanks for reporting the error,
Marius


On 15 March 2016 at 19:25, Marius van den Beek <m.vandenb...@gmail.com>
wrote:

> Yes, `git checkout release_16.01` should do it, but wait a second, there
> might actually be a problem,
> since ` File 
> "/usr/local/galaxy-prod/lib/galaxy/visualization/data_providers/genome.py",
> line 892, in get_iterator`,
> while the PR addresses 354-356!
>
>
> On 15 March 2016 at 19:23, Lubos Klucar <klu...@embnet.sk> wrote:
>
>>
>>
>> On 15. 3. 2016 19:15, Marius van den Beek wrote:
>>
>>> Hi Lubos,
>>>
>>> hmm, I'm surprised about this.
>>> Just to make sure, you are on the release_16.01 branch?
>>> (`git branch` should highlight release_16.01)
>>> This PR hasn't been merged into dev yet.
>>>
>>>
>> Hi Marius,
>>
>> it seems I need more help with git, not the Galaxy!. ;-) My git branch
>> gives me:
>>
>>   dev
>> * master
>>
>> How should I 'switch' to 16.01??..
>>
>> regards
>>
>> Lubos
>>
>> Cheers,
>>> Marius
>>>
>>> On 15 March 2016 at 19:08, Lubos Klucar <klu...@embnet.sk
>>> <mailto:klu...@embnet.sk>> wrote:
>>>
>>> Dear Marius,
>>>
>>> thanks for help, but it looks our galaxy is up to date:
>>>
>>> remote: Counting objects: 111, done.
>>> remote: Compressing objects: 100% (23/23), done.
>>> remote: Total 111 (delta 82), reused 75 (delta 75), pack-reused 13
>>> Receiving objects: 100% (111/111), 39.85 KiB, done.
>>> Resolving deltas: 100% (82/82), completed with 29 local objects.
>>>  >From https://github.com/galaxyproject/galaxy
>>> 8423fe7..23c3300  dev-> origin/dev
>>> 5a50a46..65a5b08  release_16.01 -> origin/release_16.01
>>> Already up-to-date.
>>>
>>> After Galaxy restart the problem still persist...
>>>
>>> all the best
>>>
>>> Lubos
>>>
>>> On 15. 3. 2016 18:59, Marius van den Beek wrote:
>>>
>>> Hello Lubos,
>>>
>>> this should be fixed with
>>> https://github.com/galaxyproject/galaxy/pull/1897.
>>> If you pull in the latest changes (git pull) on the
>>> release_16.01 branch
>>> things should work again.
>>>
>>> Cheers,
>>> Marius
>>>
>>> On 15 March 2016 at 18:54, Lubos Klucar <klu...@embnet.sk
>>> <mailto:klu...@embnet.sk>
>>> <mailto:klu...@embnet.sk <mailto:klu...@embnet.sk>>> wrote:
>>>
>>>  Hi,
>>>
>>>  after upgrading from 15.10 to 16.01, the trackster
>>> visualisation
>>>  tool is partly defective. It can successfully show e.g. GFF
>>> tracks,
>>>  but after adding e.g. BAM track nothing is shown. The only
>>>  suspicious debug output I could get when loading
>>> non-working-track is:
>>>
>>>  galaxy.webapps.galaxy.api.datasets ERROR 2016-03-15
>>> 11:44:38,853
>>>  Error in dataset API at listing contents: Expected bytes,
>>> got
>>>  unicode: Expected bytes, got unicode
>>>  Traceback (most recent call last):
>>> File
>>>
>>>
>>> "/usr/local/galaxy-prod/lib/galaxy/webapps/galaxy/api/datasets.py",
>>>  line 66, in show
>>>   rval = self._data( trans, dataset, **kwd )
>>> File
>>>
>>>
>>> "/usr/local/galaxy-prod/lib/galaxy/webapps/galaxy/api/datasets.py",
>>>  line 235, in _data
>>>   ref_seq=region, mean_depth=mean_depth, **kwargs )
>>> File
>>>
>>>
>>> "/usr/local/galaxy-prod/lib/galaxy/visualization/data_providers/genome.py",
>>>  line 192, in get_data
>>>   

Re: [galaxy-dev] Trackster in 16.01 broken

[Marius van den Beek]

Hello Lubos,

this should be fixed with https://github.com/galaxyproject/galaxy/pull/1897.
If you pull in the latest changes (`git pull`) on the release_16.01 branch
things should work again.

Cheers,
Marius

On 15 March 2016 at 18:59, Marius van den Beek <m.vandenb...@gmail.com>
wrote:

> Hello Lubos,
>
> this should be fixed with
> https://github.com/galaxyproject/galaxy/pull/1897.
> If you pull in the latest changes (git pull) on the release_16.01 branch
> things should work again.
>
> Cheers,
> Marius
>
> On 15 March 2016 at 18:54, Lubos Klucar <klu...@embnet.sk> wrote:
>
>> Hi,
>>
>> after upgrading from 15.10 to 16.01, the trackster visualisation tool is
>> partly defective. It can successfully show e.g. GFF tracks, but after
>> adding e.g. BAM track nothing is shown. The only suspicious debug output I
>> could get when loading non-working-track is:
>>
>> galaxy.webapps.galaxy.api.datasets ERROR 2016-03-15 11:44:38,853 Error in
>> dataset API at listing contents: Expected bytes, got unicode: Expected
>> bytes, got unicode
>> Traceback (most recent call last):
>>   File
>> "/usr/local/galaxy-prod/lib/galaxy/webapps/galaxy/api/datasets.py", line
>> 66, in show
>> rval = self._data( trans, dataset, **kwd )
>>   File
>> "/usr/local/galaxy-prod/lib/galaxy/webapps/galaxy/api/datasets.py", line
>> 235, in _data
>> ref_seq=region, mean_depth=mean_depth, **kwargs )
>>   File
>> "/usr/local/galaxy-prod/lib/galaxy/visualization/data_providers/genome.py",
>> line 192, in get_data
>> iterator = self.get_iterator( data_file, chrom, start, end, **kwargs )
>>   File
>> "/usr/local/galaxy-prod/lib/galaxy/visualization/data_providers/genome.py",
>> line 892, in get_iterator
>> data = data_file.fetch( start=start, end=end, reference=chrom )
>>   File "pysam/calignmentfile.pyx", line 868, in
>> pysam.calignmentfile.AlignmentFile.fetch (pysam/calignmentfile.c:10170)
>>   File "pysam/calignmentfile.pyx", line 787, in
>> pysam.calignmentfile.AlignmentFile.parse_region
>> (pysam/calignmentfile.c:9605)
>>   File "pysam/calignmentfile.pyx", line 1576, in
>> pysam.calignmentfile.AlignmentFile.gettid (pysam/calignmentfile.c:16665)
>>   File "pysam/calignmentfile.pyx", line 640, in
>> pysam.calignmentfile.AlignmentFile.get_tid (pysam/calignmentfile.c:8208)
>>   File "pysam/cutils.pyx", line 106, in pysam.cutils.force_bytes
>> (pysam/cutils.c:2170)
>> TypeError: Expected bytes, got unicode
>>
>> Any help to solve this problem would be highly appreciated!
>>
>> many thanks
>> --
>>
>> Lubos Klucar
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>  https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>  http://galaxyproject.org/search/mailinglists/
>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Trackster in 16.01 broken

[Marius van den Beek]

Hello Lubos,

this should be fixed with https://github.com/galaxyproject/galaxy/pull/1897.
If you pull in the latest changes (git pull) on the release_16.01 branch
things should work again.

Cheers,
Marius

On 15 March 2016 at 18:54, Lubos Klucar  wrote:

> Hi,
>
> after upgrading from 15.10 to 16.01, the trackster visualisation tool is
> partly defective. It can successfully show e.g. GFF tracks, but after
> adding e.g. BAM track nothing is shown. The only suspicious debug output I
> could get when loading non-working-track is:
>
> galaxy.webapps.galaxy.api.datasets ERROR 2016-03-15 11:44:38,853 Error in
> dataset API at listing contents: Expected bytes, got unicode: Expected
> bytes, got unicode
> Traceback (most recent call last):
>   File "/usr/local/galaxy-prod/lib/galaxy/webapps/galaxy/api/datasets.py",
> line 66, in show
> rval = self._data( trans, dataset, **kwd )
>   File "/usr/local/galaxy-prod/lib/galaxy/webapps/galaxy/api/datasets.py",
> line 235, in _data
> ref_seq=region, mean_depth=mean_depth, **kwargs )
>   File
> "/usr/local/galaxy-prod/lib/galaxy/visualization/data_providers/genome.py",
> line 192, in get_data
> iterator = self.get_iterator( data_file, chrom, start, end, **kwargs )
>   File
> "/usr/local/galaxy-prod/lib/galaxy/visualization/data_providers/genome.py",
> line 892, in get_iterator
> data = data_file.fetch( start=start, end=end, reference=chrom )
>   File "pysam/calignmentfile.pyx", line 868, in
> pysam.calignmentfile.AlignmentFile.fetch (pysam/calignmentfile.c:10170)
>   File "pysam/calignmentfile.pyx", line 787, in
> pysam.calignmentfile.AlignmentFile.parse_region
> (pysam/calignmentfile.c:9605)
>   File "pysam/calignmentfile.pyx", line 1576, in
> pysam.calignmentfile.AlignmentFile.gettid (pysam/calignmentfile.c:16665)
>   File "pysam/calignmentfile.pyx", line 640, in
> pysam.calignmentfile.AlignmentFile.get_tid (pysam/calignmentfile.c:8208)
>   File "pysam/cutils.pyx", line 106, in pysam.cutils.force_bytes
> (pysam/cutils.c:2170)
> TypeError: Expected bytes, got unicode
>
> Any help to solve this problem would be highly appreciated!
>
> many thanks
> --
>
> Lubos Klucar
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] ERROR: PIP is not a legl parament of an Ansible Play

[Marius van den Beek]

The list of tools installed on usegalaxy.org is conviniently desposited
here:
https://github.com/galaxyproject/usegalaxy-playbook/blob/master/files/galaxy/usegalaxy.org/tool_list.yaml

Normally you should be able to use this command outlined above to get all
the tools that usegalaxy.org has,
including the dependencies.

PS. please keep all traffic on the mailing list

On 4 March 2016 at 19:59, Marius van den Beek <m.vandenb...@gmail.com>
wrote:

> There is no real step by step guide, but the script itself has a help text:
>
> $ python install_tool_shed_tools.py -h
>
> Also make sure you have bioblend installed (pip install bioblend)
> It does more things than just installing tools. To just install tools you
> would use a command line like this:
>
>
> python install_tool_shed_tools.py -a "your new admin api key" -g "your new 
> galaxy instance url" -t tool_list.yml
>
> tool_list.yml follows the format outlined here:
>
> https://github.com/galaxyproject/ansible-galaxy-tools/blob/master/files/tool_list.yaml.sample
>
> Cheers,
> Marius
> ​
>
> On 4 March 2016 at 19:11, William Ge <w...@centrilliontech.com> wrote:
>
>> Thank you both, Marius and Dannon!
>>
>> Since I am not familiar with ansible, I will just run the python script.
>> I am reading about the usage of the script, trying to figure out what
>> configuration needs  to be in place and what options to use. Is there  a
>> step-by-step guide for using this script somewhere?
>>
>>
>> Best regards,
>> Bill
>>
>> --
>> *From:* Marius van den Beek <m.vandenb...@gmail.com>
>> *Sent:* Friday, March 4, 2016 7:25 AM
>> *To:* Dannon Baker
>> *Cc:* William Ge; galaxy-dev@lists.galaxyproject.org
>> *Subject:* Re: [galaxy-dev] ERROR: PIP is not a legl parament of an
>> Ansible Play
>>
>> Hi Bill,
>>
>> I think you're trying to use the ansible-galaxy-tools role as a playbook:
>> This won't work, you can only include the role in your playbook,
>> or use this playbook https://github.com/afgane/galaxy-tools-playbook
>> <https://github.com/afgane/galaxy-tools-playbook>
>> afgane/galaxy-tools-playbook · GitHub
>> <https://github.com/afgane/galaxy-tools-playbook>
>> github.com
>> README.md A ready-to-use Ansible playbook for the Galaxy Tools role.
>> Before you can use this playbook, you need to install Ansible: $ pip
>> install ansible
>> You can also directly use the python script in
>> https://github.com/galaxyproject/ansible-galaxy-tools/blob/master/files/install_tool_shed_tools.py
>> if you're more comfortable not using ansible.
>>
>> Cheers,
>> Marius
>>
>> On 4 March 2016 at 13:43, Dannon Baker <dannon.ba...@gmail.com> wrote:
>>
>>> I haven't seen this before but did a little googling.  What version is
>>> your ansible installation?   If it's a little bit older, (
>>> https://github.com/ansible/ansible/issues/5412) might be what you're
>>> seeing.
>>>
>>> It's a bit of a long read, but it looks like reinstalling or updating
>>> your ansible (or installing a new one in a fresh virtualenv for isolation)
>>> should resolve it.
>>>
>>> Let me know if that doesn't fix it up for you.
>>>
>>> On Thu, Mar 3, 2016 at 10:08 PM, William Ge <w...@centrilliontech.com>
>>> wrote:
>>>
>>>> Hi, all,
>>>>
>>>>
>>>>
>>>> Sorry to bother you again. I am trying to install tools using the
>>>> script. I have downloaded the playbook, but stuck here:
>>>>
>>>>
>>>>
>>>> [galaxy@Pegasus ansible-galaxy-tools]$ ansible-playbook
>>>> tasks/tools.yml -i "localhost," --extra-vars galaxy_tools_api_key=xcd2Glx
>>>>
>>>> ERROR: pip is not a legal parameter of an Ansible Play
>>>>
>>>>
>>>>
>>>> I did not find anything related on the Galaxy search either. Any help
>>>> will be appreciated.
>>>>
>>>>
>>>>
>>>> The web Admin interface only allows tool installation one by one. I
>>>> wish it allows batch installation there.
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Bill
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ___
>>>> Please keep all replies on the list by using "reply all"
>>>> in your m

Re: [galaxy-dev] ERROR: PIP is not a legl parament of an Ansible Play

[Marius van den Beek]

There is no real step by step guide, but the script itself has a help text:

$ python install_tool_shed_tools.py -h

Also make sure you have bioblend installed (pip install bioblend)
It does more things than just installing tools. To just install tools you
would use a command line like this:


python install_tool_shed_tools.py -a "your new admin api key" -g "your
new galaxy instance url" -t tool_list.yml

tool_list.yml follows the format outlined here:
https://github.com/galaxyproject/ansible-galaxy-tools/blob/master/files/tool_list.yaml.sample

Cheers,
Marius
​

On 4 March 2016 at 19:11, William Ge <w...@centrilliontech.com> wrote:

> Thank you both, Marius and Dannon!
>
> Since I am not familiar with ansible, I will just run the python script. I
> am reading about the usage of the script, trying to figure out what
> configuration needs  to be in place and what options to use. Is there  a
> step-by-step guide for using this script somewhere?
>
>
> Best regards,
> Bill
>
> --
> *From:* Marius van den Beek <m.vandenb...@gmail.com>
> *Sent:* Friday, March 4, 2016 7:25 AM
> *To:* Dannon Baker
> *Cc:* William Ge; galaxy-dev@lists.galaxyproject.org
> *Subject:* Re: [galaxy-dev] ERROR: PIP is not a legl parament of an
> Ansible Play
>
> Hi Bill,
>
> I think you're trying to use the ansible-galaxy-tools role as a playbook:
> This won't work, you can only include the role in your playbook,
> or use this playbook https://github.com/afgane/galaxy-tools-playbook
> <https://github.com/afgane/galaxy-tools-playbook>
> afgane/galaxy-tools-playbook · GitHub
> <https://github.com/afgane/galaxy-tools-playbook>
> github.com
> README.md A ready-to-use Ansible playbook for the Galaxy Tools role.
> Before you can use this playbook, you need to install Ansible: $ pip
> install ansible
> You can also directly use the python script in
> https://github.com/galaxyproject/ansible-galaxy-tools/blob/master/files/install_tool_shed_tools.py
> if you're more comfortable not using ansible.
>
> Cheers,
> Marius
>
> On 4 March 2016 at 13:43, Dannon Baker <dannon.ba...@gmail.com> wrote:
>
>> I haven't seen this before but did a little googling.  What version is
>> your ansible installation?   If it's a little bit older, (
>> https://github.com/ansible/ansible/issues/5412) might be what you're
>> seeing.
>>
>> It's a bit of a long read, but it looks like reinstalling or updating
>> your ansible (or installing a new one in a fresh virtualenv for isolation)
>> should resolve it.
>>
>> Let me know if that doesn't fix it up for you.
>>
>> On Thu, Mar 3, 2016 at 10:08 PM, William Ge <w...@centrilliontech.com>
>> wrote:
>>
>>> Hi, all,
>>>
>>>
>>>
>>> Sorry to bother you again. I am trying to install tools using the
>>> script. I have downloaded the playbook, but stuck here:
>>>
>>>
>>>
>>> [galaxy@Pegasus ansible-galaxy-tools]$ ansible-playbook tasks/tools.yml
>>> -i "localhost," --extra-vars galaxy_tools_api_key=xcd2Glx
>>>
>>> ERROR: pip is not a legal parameter of an Ansible Play
>>>
>>>
>>>
>>> I did not find anything related on the Galaxy search either. Any help
>>> will be appreciated.
>>>
>>>
>>>
>>> The web Admin interface only allows tool installation one by one. I wish
>>> it allows batch installation there.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Bill
>>>
>>>
>>>
>>>
>>>
>>> ___
>>> Please keep all replies on the list by using "reply all"
>>> in your mail client.  To manage your subscriptions to this
>>> and other Galaxy lists, please use the interface at:
>>>   https://lists.galaxyproject.org/
>>>
>>> To search Galaxy mailing lists use the unified search at:
>>>   http://galaxyproject.org/search/mailinglists/
>>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] ERROR: PIP is not a legl parament of an Ansible Play

[Marius van den Beek]

Hi Bill,

I think you're trying to use the ansible-galaxy-tools role as a playbook:
This won't work, you can only include the role in your playbook,
or use this playbook https://github.com/afgane/galaxy-tools-playbook
You can also directly use the python script in
https://github.com/galaxyproject/ansible-galaxy-tools/blob/master/files/install_tool_shed_tools.py
if you're more comfortable not using ansible.

Cheers,
Marius

On 4 March 2016 at 13:43, Dannon Baker  wrote:

> I haven't seen this before but did a little googling.  What version is
> your ansible installation?   If it's a little bit older, (
> https://github.com/ansible/ansible/issues/5412) might be what you're
> seeing.
>
> It's a bit of a long read, but it looks like reinstalling or updating your
> ansible (or installing a new one in a fresh virtualenv for isolation)
> should resolve it.
>
> Let me know if that doesn't fix it up for you.
>
> On Thu, Mar 3, 2016 at 10:08 PM, William Ge 
> wrote:
>
>> Hi, all,
>>
>>
>>
>> Sorry to bother you again. I am trying to install tools using the script.
>> I have downloaded the playbook, but stuck here:
>>
>>
>>
>> [galaxy@Pegasus ansible-galaxy-tools]$ ansible-playbook tasks/tools.yml
>> -i "localhost," --extra-vars galaxy_tools_api_key=xcd2Glx
>>
>> ERROR: pip is not a legal parameter of an Ansible Play
>>
>>
>>
>> I did not find anything related on the Galaxy search either. Any help
>> will be appreciated.
>>
>>
>>
>> The web Admin interface only allows tool installation one by one. I wish
>> it allows batch installation there.
>>
>>
>>
>> Thanks,
>>
>> Bill
>>
>>
>>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/
>>
>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Docker tools

[Marius van den Beek]

Hello Abdulrahman,

I believe you can control the docker host in the job_conf.xml.
https://github.com/galaxyproject/galaxy/blob/dev/config/job_conf.xml.sample_advanced#L227
I haven't' tried this yet, but it seems that it should work.

Cheers,
Marius

On 2 March 2016 at 09:55, Abdulrahman Azab  wrote:

> Hi Galaxy,
>
>
> I'm working on a project for running galaxy on the top of a docker swarm
> cluster. The docker tag in the tool xml so far supports access only to the
> local docker host. To submit to the swarm manager, the command should be:
>
>
> $ docker -H tcp://: run ...
>
> instead of:
>
> $ docker run ...
>
>
> Is it possible to add an xml tag attribute to support that option?
>
> This would off course require that all files are located in a shared
> file-system.
>
>
> Thank you,
>
> Yours sincerely,
> Abdulrahman Azab
>
> Head engineer
> ELIXIR.NO, Genomic Hyperbrowser Team
> Research Group for Biomedical Informatics, Department of Informatics (IFI)
> Research Support Services Group, University Center for Information
> Technology (USIT)
> University of Oslo, Boks 1072 Blindern, NO-0316 OSLO, Norway
> Email: a...@ifi.uio.no, a...@usit.uio.no
> Cell-phone: +47 46797339
> 
> Senior Lecturer in Computer Engineering
> Faculty of Engineering, University of Mansoura, 35516-Mansoura, Egypt
> Email: abdulrahman.a...@mans.edu.eg
> 
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Difficulty uploading BAM files

[Marius van den Beek]

Hello Scott,

you do need a samtools binary on GALAXY's PATH.
This is not samtools installed from the toolshed.

Best,
Marius

On 26 February 2016 at 18:49, Scott Szakonyi 
wrote:

> Hi,
>
> I'm having trouble uploading BAM files in our development environment.
> Other files types are uploading without issue. I'm testing with a small BAM
> file I downloaded, and I'm able to successfully upload it on our production
> server and usegalaxy.org. When I try to upload in the development
> environment, I get the following error:
>
> Traceback (most recent call last):
>   File "/vectorbase/web/Galaxy/galaxy/tools/data_source/upload.py", line
> 431, in 
> __main__()
>   File "/vectorbase/web/Galaxy/galaxy/tools/data_source/upload.py", line
> 420, in __main__
> add_file( dataset, registry, json_file, output_path )
>   File "/vectorbase/web/Galaxy/galaxy/tools/data_source/upload.py", line
> 347, in add_file
> if link_data_only == 'copy_files' and
> datatype.dataset_content_needs_grooming( output_path ):
>   File "/vectorbase/web/Galaxy/galaxy/lib/galaxy/datatypes/binary.py",
> line 218, in dataset_content_needs_grooming
> version = self._get_samtools_version()
>   File "/vectorbase/web/Galaxy/galaxy/lib/galaxy/datatypes/binary.py",
> line 179, in _get_samtools_version
> raise Exception(message)
> Exception: Attempting to use functionality requiring samtools, but it
> cannot be located on Galaxy's PATH.
>
> I've checked my installed tools, and all the same Samtools packages are
> successfully installed on the development environment as in production. I
> don't see any missing dependencies or anything like that. At this point I'm
> stumped. If anyone can offer some guidance on how to resolve this, I'd be
> most appreciative.
>
> Best regards,
>
> --
> Scott B. Szakonyi
> Research Programmer
>
> *Center for Research Computing*
> 107 Information Technology Center
> Notre Dame, IN 46556
> http://crc.nd.edu
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Collapse tool panel and history panel

[Marius van den Beek]

Hi Makis,

if you change the javascript in client/ ... you need to run grunt.
The instructions are in
https://github.com/galaxyproject/galaxy/blob/dev/client/README.md.

Cheers,
Marius

On 25 February 2016 at 10:57, Makis Ladoukakis 
wrote:

> Just an update/bump of the issue. With the new 16.01 version I made the
> changes in the /client/galaxy/scripts/apps/analysis.js
> b/client/galaxy/scripts/apps/analysis.js file but the none of the panels
> were collapsed even when i restarted my Galaxy instance.
>
> I don't really get the mechanics behind it. Seems like any changes i do
> either in the js files or the mako scripts never take effect in the actual
> instance. I even tried to delete everything in there leaving just a blank
> file to test my theory and after restart the Galaxy instance initializes
> with no errors at all.
>
> Any ideas?
> Makis
>
> --
> From: makis4e...@hotmail.com
> To: dannon.ba...@gmail.com
> Date: Mon, 22 Feb 2016 13:47:59 +0200
> CC: galaxy-...@lists.bx.psu.edu
>
> Subject: Re: [galaxy-dev] Collapse tool panel and history panel
>
> The git log ..origin/master command produced no output so I guess I have
> the latest release (15.10.1 maybe?). The git pull command confirmed that.
> Should I use a release from the dev branch?
>
> Kind regards,
> Makis
>
> --
> Date: Fri, 19 Feb 2016 08:39:35 -0500
> Subject: Re: [galaxy-dev] Collapse tool panel and history panel
> From: dannon.ba...@gmail.com
> To: makis4e...@hotmail.com
> CC: galaxy-...@lists.bx.psu.edu
>
> Sure, I can help.  Which version of Galaxy are you currently running?  I
> didn't check the history very well and the fix I suggested will work for
> the forthcoming 16.01 release (or current dev branch), but not 15.10.
>
>
> On Fri, Feb 19, 2016 at 8:32 AM, Makis Ladoukakis 
> wrote:
>
> Hello Dannon,
>
> Sorry for asking but can you help me out a little bit with that? I
> searched in my galaxy directory in order to add those two lines you mention
> in the link but there is no galaxy/scripts/apps/analysis.js file and
> neither any file containing the string 'analysisPage.right.historyView'.
>
> Any advice?
>
> Thank you,
> Makis
> --
> Date: Mon, 28 Dec 2015 14:30:02 -0500
> Subject: Re: [galaxy-dev] Collapse tool panel and history panel
> From: dannon.ba...@gmail.com
> To: makis4e...@hotmail.com
> CC: galaxy-...@lists.bx.psu.edu
>
>
> Hi Makis,
>
> We've restructured much of the client code fairly recently.  Something
> like this should work for you now:
>
> https://gist.github.com/dannon/bd470d9c70019b07cb8b
>
>
> Sorry for the slow response!
>
> -Dannon
>
>
> On Sat, Dec 19, 2015 at 2:57 PM, Makis Ladoukakis 
> wrote:
>
> Shamelessly bumping my own question... Does anyone have any ideas about
> collapsing the history and tool panel?
>
> Thank you,
> Makis Ladoukakis
>
> --
> From: makis4e...@hotmail.com
> To: galaxy-...@lists.bx.psu.edu
> Date: Wed, 16 Dec 2015 13:41:19 +0200
> Subject: [galaxy-dev] Collapse tool panel and history panel
>
>
> Hello,
>
> Does anyone know how to start my Galaxy instance with the history panel
> and tool panel collapsed by default?
>
> I used to do it by editing the galaxy/templates/base/base_panels.mako
> script and adding
>
> *lp.do_toggle();*
>
> and
>
>
>
> *rp.do_toggle();*inside the if sections of *%if self.has_left_panel: *and *%if
> self.has_right_panel: *respectively but with my new installed instance
> this doesn't seem to work anymore.
>
> Any ideas anyone?
>
> Thank you,
> Makis Ladoukakis
>
>
>
>
> ___ Please keep
> all replies on the list by using "reply all" in your mail client. To manage
> your subscriptions to this and other Galaxy lists, please use the interface
> at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use
> the unified search at: http://galaxyproject.org/search/mailinglists/
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
>
>
>
> ___ Please keep
> all replies on the list by using "reply all" in your mail client. To manage
> your subscriptions to this and other Galaxy lists, please use the interface
> at: https://lists.galaxyproject.org/ To search Galaxy mailing lists use
> the unified search at: http://galaxyproject.org/search/mailinglists/
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> 

Re: [galaxy-dev] Docker and alpine linux

[Marius van den Beek]

Just my 2 cents about this:
I feel your pain, our connection to the docker hub is horrible, it takes an
hour to pull the GIE images ... .
(While it only takes seconds from the cloud ...).
The galaxy docker images are pretty fat, because we use them VM-style, in
principle nothing wrong about that.
So the base-image is about 1.2GB  add a few tools and you're quickly
reaching 5GB.
If in addition you use interactive environments ... add a few GB more.

I think that instead of trying to reduce the size of the base-image, it
might be a better effort to separate the components
into a proxy-image, database image (perhaps 2, one for tools / one for
user-data), galaxy-image,
cluster-image ... and so on. This would allow you to just update the tools
and galaxy image regularly,
plus you could do all the neat docker stuff, like versioning, committing,
rolling updates, streaming database replication, worker scaling ...
It's certainly something I would be interested in.
The other thing is to make sure that the tool dependencies are as slim as
possible. Having many different R packages
and their source lying around makes for a lot of data. Hopefully conda can
alleviate that situation.

Cheers,
Marius

On 23 February 2016 at 09:36, Björn Grüning 
wrote:

> Hi Tiago,
>
> thanks for the heads-up. I also tried alpine some time ago but Galaxy
> needs some external dependencies which Nate is building also with a ppa
> for Debian based systems. So this is not so easy to migrate.
>
> What we could do is to orchestrate the containers and its deps:
>
> https://github.com/bgruening/docker-galaxy-stable/issues/43
>
> But this has the big disadvantages of not sharing the setup with other
> Galaxy installations, like the VM installation from planemo-machine.
>
> So in the end I stoped to make it more modular and tried to share as
> much as possible with other installations of Galaxy and move more and
> more into the ansible-playbook.
>
> Thanks Tiago for trying this,
> Bjoern
>
> Am 23.02.2016 um 05:15 schrieb Tiago Antao:
> > Dear all,
> >
> > This email is mostly to report a negative result, maybe to help others
> > _not_ trying something.
> >
> > I researched the possibility of replacing ubuntu with alpine on the
> > Docker images (alpine seems to be used more and more on docker images,
> > with plenty of official containers now based on it and not on debian).
> >
> > The reason alpine is used, its because it generates very small
> > containers (a bare bones one is below 10 MB). But, due the the large
> > dependencies of galaxy, the gain is negligible. Maybe 200 MB or so. 20%
> > is something, but not a revolution.
> >
> > This being said, for servers with smaller dependencies (a mail server,
> > web, ldap, dns...) alpine really reduces the footprint of docker
> > containers.
> >
> > Tiago
> >
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] JBrowse on dev branch: progress

[Marius van den Beek]

I have occasionally seen this.
GNU-coreutils hangs while compiling on ubuntu 14.04 ... there is a
`nanosleep` in the configure step,
which is never returning.
"checking for working nanosleep..."
https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/1079889
Just kill the process (this will happen twice), and it will work.
I will have a look if this is still happening with newer versions of
coreutils.

On 22 February 2016 at 10:43, Peter van Heusden  wrote:

> Ok, second attempt, perl install died with the same error, so I suspect
> you're right about the cause. Eric?
>
> Peter
>
> On 22 February 2016 at 11:41, Peter van Heusden  wrote:
>
>> Thanks Peter
>>
>> I managed, on the second attempt, to get past the gnu coreutils problem.
>> Zipho noticed a weird lockup though: conftest that seemed to be from the
>> gnu coreutils install wanted to listen on port 4001, which is the same port
>> our uwsgi server listens on. The install has now completed but I'll try and
>> find time to go back and verify this.
>>
>> On the second attempt perl seems to be installing, so I'll see if it
>> completes or hits the error you're mentioning.
>>
>> Peter
>>
>> On 22 February 2016 at 11:34, Peter Briggs > > wrote:
>>
>>> Hi Peter
>>>
>>> I think that package_perl_5_18 in the tool shed is broken, I came across
>>> the same error last week on both our production instance and a local 'test'
>>> instance (both v15.10).
>>>
>>> My suspicion is that the following lines are to blame:
>>>
>>> ...
>>> >> sha256sum="5c69adb47ab828aa3e8b5be89b88cd49c6a0d0dae2e8b3bca17a9ce699190e7b"
>>> type="download_file">
>>> http://www.cpan.org/authors/id/A/AP/APEIRON/local-lib-1.008009.tar.gz
>>> 
>>> tar xfvz
>>> local-lib-1.008009.tar.gz
>>> ...
>>>
>>> Looking at the Galaxy internals I believe that the 'download_file'
>>> action automatically unpacks the archive and cd's into the resulting
>>> directory, hence the failure for the subsequent 'tar xfvz ...' command. But
>>> I haven't had the time yet to verify this or create a patch to see if the
>>> removal of this line addresses the problem.
>>>
>>> I don't think I've seen the problem with 'package_gnu_coreutils_8_22'
>>> that you describe however so it may be that my hypothesis is not correct.
>>>
>>> HTH
>>>
>>> Best wishes
>>>
>>> Peter
>>>
>>>
>>> On 22/02/16 05:10, Peter van Heusden wrote:
>>>
 Hi there Eric and others

 I tried to install the JBrowse tool on a dev branch server running on an
 Ubuntu 14.04 VM.

 It got stuck at the package_gnu_coreutils_8_22 dependency (revision
 ac64dfe4b1fb) - this ended up in "Installing tool dependencies" state.

 If I go look at the tool details it highlights errors
 in package_perl_5_18 which in turn refers to errors in gnu_coreutils
 (just "Error" state) and then this more informative failure for perl:

 "tar (child): local-lib-1.008009.tar.gz: Cannot open: No such file or
 directory
 tar (child): Error is not recoverable: exiting now
 tar: Child returned status 2
 tar: Error is not recoverable: exiting now"

 I wonder if this is a sign of a download error?

 Digging a bit more I find the error "Shutting down process id 21094
 because it generated no output for the defined timeout period of 3600.0
 seconds." for gnu_coreutils.

 I'm trying to do a reinstall of the gnu_coreutils and perl packages to
 see if I can get further.

 Peter


 ___
 Please keep all replies on the list by using "reply all"
 in your mail client.  To manage your subscriptions to this
 and other Galaxy lists, please use the interface at:
https://lists.galaxyproject.org/

 To search Galaxy mailing lists use the unified search at:
http://galaxyproject.org/search/mailinglists/


>>> --
>>> Peter Briggs peter.bri...@manchester.ac.uk
>>> Bioinformatics Core Facility University of Manchester
>>> B.1083 Michael Smith Bldg Tel: (0161) 2751482
>>>
>>
>>
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Importing workflows via command-line/Docker script

[Marius van den Beek]

Hi Cam,

this would be possible using bioblend (
https://bioblend.readthedocs.org/en/latest/api_docs/galaxy/all.html?highlight=workflow#module-bioblend.galaxy.workflows).
Take a look at the `import_workflow_from_local_path` function.
You'll need to have galaxy running for this (same as for installing tools).

Cheers,
Marius


On 27 January 2016 at 03:14, Cameron Jack  wrote:

> Hi all,
>
>
> I've been developing a Dockerfile and repositories to enable users to set
> up an NGS forensics pipeline. So far so good, I can build the image (based
> on Bjoern Gruening's excellent galaxy-stable Docker image) and incorporate
> all the various tools I need, including custom ones with custom interfaces.
> I've also built some workflows within a running image and exported them to
> .ga files, but now I'm wondering if there is a way to import those back in
> to a new image via the Dockerfile? I'd rather have everything set up for
> the users upon building the image, rather than logging in as admin and then
> importing the workflow. Is this possible? If so, how?
>
>
> Thanks for any and all answers.
>
>
> Best regards,
>
> Cam
>
>
> Cameron Jack
>
> Bioinformatician
> ANU Bioinformatics Consulting Unit
> The Australian National University
>
> The John Curtin School of Medical Research
> Building 131 Garran Road
> Acton 0200, ACT
>
> Ph (office): +61 2 612 51128
> Ph (mobile): +61 4 2368 0745
> Email: cameron.j...@anu.edu.au
> Group email: a...@anu.edu.au
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>   https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>   http://galaxyproject.org/search/mailinglists/
>
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] gcc error during pixman-0.32.4 installation on Mac OS X 10.11

[Marius van den Beek]

Hello Hans,

I'm not sure if it would help to use the pre-compiled binaries,
as you would still need some of the underlying libraries for building
R packages. We're not far from having this fixed ... there is a newer
revision of R on the main toolshed,
that nevertheless will (probably) fail because of a missing include of what
seems to be fontconfig.
I had it working yesterday already, I'm trying to see what went wrong in
the meantime ...

Marius

On 4 December 2015 at 09:20, Hans Rudolph  wrote:

> Hi,
>
> I now learned that R-3.2.2 binaries for Mac OS X are available (
> https://cran.r-project.org/bin/macosx/) - could that be a potential help
> for R-related problems in galaxy on Mac OS X?
>
> Cheers,
> Hans
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/