from:"Martin Vickers"

Re: [galaxy-dev] Creating R galaxy wrapper where a function checks file extension

2015-10-02 Thread Martin Vickers


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Peter,

That's great, worked perfectly.

Thank you very much,

Martin

On 10/02/2015 11:26 AM, Peter Cock wrote:
> If you have a wrapper script, then you can do it there.
>
> It is also possible in the XML directly by building a composite
> command with && separators (which must be escaped as
>  or using CDATA), e.g.
>
>
https://github.com/galaxyproject/tools-devteam/blob/master/tools/samtool_filter2/samtool_filter2.xml
> (nice BAM and BAI example)
>
>
https://github.com/galaxyproject/tools-devteam/blob/master/tools/freebayes/freebayes.xml
> (does a lot of symlinks)
>
>
https://github.com/galaxyproject/tools-iuc/blob/master/tools/bedtools/multiCov.xml
> (part of a for loop for multiple BAM and BAI files)
>
> Note you can't use  in this
> style as the first command is the shell command ln.
>
> Peter
>
>
> On Fri, Oct 2, 2015 at 11:17 AM, Martin Vickers <mj...@aber.ac.uk> wrote:
>>
> Hi Peter,
>
> Thanks for the reply. Are there any examples of this in action? Is it done
> in the xml wrapper somewhere or do I need to write a script that does this
> and then call the Rscript?
>
> Cheers,
>
> Martin
>
> On 10/02/2015 11:13 AM, Peter Cock wrote:
> >>> Creating symlinks with normal extensions is a common
> >>> approach here where the Galaxy filenames won't work.
> >>>
> >>> Peter
> >>>
> >>> On Fri, Oct 2, 2015 at 10:57 AM, Martin Vickers <mj...@aber.ac.uk>
wrote:
> >>>>
> >>> Hi all,
> >>>
> >>> I've been tasked with getting someone's R script working in our galaxy
> >>> installation and I'm struggling to resolve an issue with an
bioconductor
> >>> function that appears to not like the galaxy naming convention (e.g.
> >>> dataset_2.dat) for the bam index file.
> >>>
> >>> The R script can run from the command line like this;
> >>>
> >>> Rscript script.R input1.xls alignment.bam alignment.bai p
> >>>
> >>> so I have created a wrapper using planemo that does this, and
that's fine.
> >>>
> >>> The problem is, when I run the script in galaxy, I get the
following error
> >>> from R;
> >>>
> >>> Error in value[[3L]](cond) :
> >>> failed to open BamFile: failed to load BAM index
> >>> file: /tmp/tmpdP2eBC/files/000/dataset_3.dat
> >>> Calls: f_oGRSeparateStrands ... tryCatch -> tryCatchList ->
tryCatchOne ->
> >>> 
> >>> In addition: Warning messages:
> >>> In doTryCatch(return(expr), name, parentenv, handler) :
> >>> [bam_index_load] fail to load BAM index.
> >>> Execution halted
> >>>
> >>> The function that reads the bam + bai file in the script is;
> >>>
> >>> bam = readGAlignments(file=bam.file, index=bai.file,
> >>> param=ScanBamParam(which=gr.signal))
> >>>
> >>>
http://www.rdocumentation.org/packages/GenomicRanges/html/GAlignments.html
> >>>
> >>> In the documentation for this function it states the following;
> >>>
> >>> "file, index, The path to the BAM file to read, and to the index
file of
> >>> the
> >>> BAM file to read, respectively. The latter is given /without/ the
'.bai'
> >>> extension. See |scanBam
> >>>
<http://www.rdocumentation.org/packages/Rsamtools/functions/scanBam.html>|
> >>> for more information."
> >>>
> >>> I've played around with this a little and from the command line
I've been
> >>> successfully able to run the script to completion when the input
bam file
> >>> and bai are completely different names;
> >>>
> >>> e.g.
> >>>
> >>> Rscript script.R input1.xls something.bam meh.bai p
> >>>
> >>> but I can't run it if I change the extension, e.g.
> >>>
> >>> Rscript script.R input1.xls something.dat meh.dat p
> >>> Error in value[[3L]](cond) :
> >>>
> >>> failed to open BamFile: failed to load BAM index
> >>> file: /tmp/tmpdP2eBC/files/000/dataset_3.dat
> >>> Calls: f_oGRSeparateStrands ... tryCatch -> tryCatchList ->
tryCatchOne ->
> >>> 
> >>> In addition: Warning messages:
> >>> In doTryCatch(return(expr), name, parentenv, handler) :
> >>> [bam_index_load] fail to load BAM index.
> >>> Execution halted

[galaxy-dev] Creating R galaxy wrapper where a function checks file extension

2015-10-02 Thread Martin Vickers


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi all,

I've been tasked with getting someone's R script working in our galaxy
installation and I'm struggling to resolve an issue with an bioconductor
function that appears to not like the galaxy naming convention (e.g.
dataset_2.dat) for the bam index file.

The R script can run from the command line like this;

Rscript script.R input1.xls alignment.bam alignment.bai p

so I have created a wrapper using planemo that does this, and that's fine.

The problem is, when I run the script in galaxy, I get the following
error from R;

Error in value[[3L]](cond) :
failed to open BamFile: failed to load BAM index
file: /tmp/tmpdP2eBC/files/000/dataset_3.dat
Calls: f_oGRSeparateStrands ... tryCatch -> tryCatchList -> tryCatchOne
-> 
In addition: Warning messages:
In doTryCatch(return(expr), name, parentenv, handler) :
[bam_index_load] fail to load BAM index.
Execution halted

The function that reads the bam + bai file in the script is;

bam = readGAlignments(file=bam.file, index=bai.file,
param=ScanBamParam(which=gr.signal))

http://www.rdocumentation.org/packages/GenomicRanges/html/GAlignments.html

In the documentation for this function it states the following;

"file, index, The path to the BAM file to read, and to the index file of
the BAM file to read, respectively. The latter is given /without/ the
'.bai' extension. See |scanBam
<http://www.rdocumentation.org/packages/Rsamtools/functions/scanBam.html>|
for more information."

I've played around with this a little and from the command line I've
been successfully able to run the script to completion when the input
bam file and bai are completely different names;

e.g.

Rscript script.R input1.xls something.bam meh.bai p

but I can't run it if I change the extension, e.g.

Rscript script.R input1.xls something.dat meh.dat p
Error in value[[3L]](cond) :

failed to open BamFile: failed to load BAM index
file: /tmp/tmpdP2eBC/files/000/dataset_3.dat
Calls: f_oGRSeparateStrands ... tryCatch -> tryCatchList -> tryCatchOne
-> 
In addition: Warning messages:
In doTryCatch(return(expr), name, parentenv, handler) :
[bam_index_load] fail to load BAM index.
Execution halted


Does anyone have any experience of resolving this kind of issue?


- -- 

- --
Dr. Martin Vickers

Data Manager/HPC Systems Administrator
Institute of Biological, Environmental and Rural Sciences
IBERS New Building
Aberystwyth University
SY23 3FG

w: http://www.martin-vickers.co.uk/
e: mj...@aber.ac.uk
t: 01970 62 2807

-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.14 (GNU/Linux)

iQEcBAEBAgAGBQJWDlUgAAoJEHa0a8GkKQgIpUcIAK9OZ9402FRaROF89b6bO1CY
d+KP/X0NCxY+AXEddU3lhXi8O9haaKRA/pmxsGyydP0VQfaCrSSlS9nzhncG6Zav
gpXe5NjFkwe8a/qaZTVf0YjlYq0sI81S7QAJnzlKXvYSjcz1xNDE9O4qSHqdhefX
oI6GWmR+vHi7CTqNuHwpyY9Qoafd5nls0oQoCYDh9YWrn9GZ8fe1qQ9W7iUli/oT
dPMYwWs0OL8NfSCuIcz2Dbr1XMt+o5HpQY/afoGWaq3zz5Ie0NhCLKz34bFps3IW
8K0DEROmCTxsXBa54gnnkhVgCI3zkfFCfa47xp13vT2dJALdzYoT6vDj8aC/mes=
=ijgr
-END PGP SIGNATURE-

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Creating R galaxy wrapper where a function checks file extension

2015-10-02 Thread Martin Vickers


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Peter,

Thanks for the reply. Are there any examples of this in action? Is it
done in the xml wrapper somewhere or do I need to write a script that
does this and then call the Rscript?

Cheers,

Martin

On 10/02/2015 11:13 AM, Peter Cock wrote:
> Creating symlinks with normal extensions is a common
> approach here where the Galaxy filenames won't work.
>
> Peter
>
> On Fri, Oct 2, 2015 at 10:57 AM, Martin Vickers <mj...@aber.ac.uk> wrote:
>>
> Hi all,
>
> I've been tasked with getting someone's R script working in our galaxy
> installation and I'm struggling to resolve an issue with an bioconductor
> function that appears to not like the galaxy naming convention (e.g.
> dataset_2.dat) for the bam index file.
>
> The R script can run from the command line like this;
>
> Rscript script.R input1.xls alignment.bam alignment.bai p
>
> so I have created a wrapper using planemo that does this, and that's fine.
>
> The problem is, when I run the script in galaxy, I get the following error
> from R;
>
> Error in value[[3L]](cond) :
> failed to open BamFile: failed to load BAM index
> file: /tmp/tmpdP2eBC/files/000/dataset_3.dat
> Calls: f_oGRSeparateStrands ... tryCatch -> tryCatchList -> tryCatchOne ->
> 
> In addition: Warning messages:
> In doTryCatch(return(expr), name, parentenv, handler) :
> [bam_index_load] fail to load BAM index.
> Execution halted
>
> The function that reads the bam + bai file in the script is;
>
> bam = readGAlignments(file=bam.file, index=bai.file,
> param=ScanBamParam(which=gr.signal))
>
> http://www.rdocumentation.org/packages/GenomicRanges/html/GAlignments.html
>
> In the documentation for this function it states the following;
>
> "file, index, The path to the BAM file to read, and to the index file
of the
> BAM file to read, respectively. The latter is given /without/ the '.bai'
> extension. See |scanBam
> <http://www.rdocumentation.org/packages/Rsamtools/functions/scanBam.html>|
> for more information."
>
> I've played around with this a little and from the command line I've been
> successfully able to run the script to completion when the input bam file
> and bai are completely different names;
>
> e.g.
>
> Rscript script.R input1.xls something.bam meh.bai p
>
> but I can't run it if I change the extension, e.g.
>
> Rscript script.R input1.xls something.dat meh.dat p
> Error in value[[3L]](cond) :
>
> failed to open BamFile: failed to load BAM index
> file: /tmp/tmpdP2eBC/files/000/dataset_3.dat
> Calls: f_oGRSeparateStrands ... tryCatch -> tryCatchList -> tryCatchOne ->
> 
> In addition: Warning messages:
> In doTryCatch(return(expr), name, parentenv, handler) :
> [bam_index_load] fail to load BAM index.
> Execution halted
>
>
> Does anyone have any experience of resolving this kind of issue?
>
>
>>
>>
>> ___
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/

- -- 

- --
Dr. Martin Vickers

Data Manager/HPC Systems Administrator
Institute of Biological, Environmental and Rural Sciences
IBERS New Building
Aberystwyth University
SY23 3FG

w: http://www.martin-vickers.co.uk/
e: mj...@aber.ac.uk
t: 01970 62 2807
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.14 (GNU/Linux)

iQEcBAEBAgAGBQJWDlmjAAoJEHa0a8GkKQgIqqkH/RwvFjF//9z3ucrAziYKbuia
qe9fFiAUhfgMFYUxVxatdR1CqzLChcONonPDgcouxGO7i1EgGNmoRqYdEdCrHbPF
WXpS/1txd8ieIxIp4IjrrozoXnd7x360M6MZd8yh6YxNd+LXQdZyWBAYUePwl5RQ
x/rwaJHnfw2hiWtHLWlrqHQJESz4zo1L0Wjymy8ehyvo9zG6760/zZSInaHmFPkN
6YJVmr0o/k1FpQrXNaRlmxMKC9jbPoDvF6ZkpseA9y1ojmFpS0mrBBLxZyaGaru9
iKH4TYnD2PTU5L+2bmCbAiEa7us0KXEx74Mq8WaKQ/hwsvXQiCpXXmvPj7oCFSI=
=MVS2
-END PGP SIGNATURE-

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] Adding data libraries from filesystem path creating duplicates

2015-09-21 Thread Martin Vickers


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi John,

Thanks for taking the time to reply. I never thought to look at the
proxy settings but I think you're right, the behaviour seems to match
what you've described.

Like you I'm not really an expert on proxies and have no idea what would
be mis-configured that would cause this.

I'm using nginx and the configuration is as described in the wiki. I've
not loaded any special extensions.

nginx is configured like this;

upstream galaxy_app {
server localhost:8090;
server localhost:8091;
server localhost:8092;
server localhost:8093;
server localhost:8094;
server localhost:8095;
}

server {
# pass to uWSGI by default
location / {
proxy_pass http://galaxy_app;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
proxy_set_header X-URL-SCHEME https;
}

static content

}

and in galaxy.ini I have a bunch of handlers, e.g.


[server:handler0]
use = egg:Paste#http
port = 8090
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

I thought that maybe the issue was to do with the 'job admin
complication' very briefly mentioned here;

https://production-galaxy-instances-with-cloudman-and-cloudbiolinux.readthedocs.org/en/latest/

so I added this to my nginx conf

location /admin/jobs {
proxy_pass  http://localhost:8090;
}


so this complication is not the one I'm having here.

Are any of the people John mentioned having this issue here on the dev
board?

Cheers,

Martin

On 09/14/2015 03:11 PM, John Chilton wrote:
> If I had to guess, I would guess this is caused by a mis-configured
> proxy (nginx or Apache) that is resubmitting a POST request that is
> taking Galaxy to long to respond to. Order of events being something
> like:
>
> - User clicks to upload library items.
> - Proxy gets requests and passes to Galaxy
> - Galaxy takes a long time to process request and doesn't respond
> within a timeout.
> - Proxy resends POST request to Galaxy.
> - Galaxy takes a long time to process request and doesn't respond
> within a timeout.
> ...
>
> Proxies should never resend POST requests to Galaxy as far a I can
> imagine, but we have seen this for instance when submitting workflows.
> Some people have had their proxy retry that request repeatedly.
>
> I don't really know if this is a problem with the default proxy
> configurations we list on the wiki or if it comes down to
> customizations or special loaded extensions at various sites that have
> encountered this.
>
> Is this enough to help debug the problem? I'm not really an expert on
> specific proxies, etc... and you have it there and seem to be able to
> reproduce the problem. If you do want further help I would post the
> proxy you are using, the extensions, the configuration, and the Galaxy
> logs corresponding to this incident to see if we can see the repeated
> posts and the route that is being posted to.
>
> If you are not using a proxy, then I am stumped :(.
>
> -John
>
>
> On Fri, Sep 4, 2015 at 12:04 PM, Martin Vickers <mj...@aber.ac.uk> wrote:
>> Hi All,
>>
>> I've noticed an issue a couple of times now where I've added a directory
>> of fastq's from an NFS mounted filesystem (reference only rather than
>> copying into galaxy) and then galaxy times out. Load average begins to
>> get really high and then consumes all the RAM and sometimes crashes.
>> These are the same symptom as I had before with this issue that was
>> never resolved;
>>
>> http://dev.list.galaxyproject.org/run-sh-segfault-td4667549.html#a4667553
>>
>> What I've noticed is that in the dataset I'm uploading to galaxy, there
>> are suddenly many duplicates. In this example that's just happened,
>> there are 288 fastq.gz files in the physical folder, but galaxy has
>> created 6 references to each file resulting in 1728 datasets in the
>> folder (see attached images).
>>
>> When this happened before and crashed the galaxy application, whenever
>> it restarted it'd try to resume what it was doing which created an
>> endless loop of retrying and crashing until the job was removed.
>>
>> Does anyone know what may be causing this?
>>
>> Cheers,
>>
>> Martin
>>
>> --
>>
>> --
>> Dr. Martin Vickers
>>
>> Data Manager/HPC Systems Administrator
>> Institute of Biological, Environmental and Rural Sciences
>> IBERS New Building
>> Aberystwyth University
>>
>> w: http://www.martin-vickers.co.uk/
>> e: mj...@aber.ac.uk
>> t: 01970 62 2807
>>
>>
>> _______
>> Please keep all replies on the

Re: [galaxy-dev] loading with Auto detect and fastqsanger options issue!

2015-09-07 Thread Martin Vickers


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Hak,

The tool is called archive_datatypes written by cmonjeau and is
available in the Galaxy Main Tool shed.

And here is the trello card to vote for support of compressed versions
of standard data types.

https://trello.com/c/3RkTDnIn/345-666-support-gzipped-gz-compressed-versions-of-standard-datatypes

and was discussed here;

https://biostar.usegalaxy.org/p/13076/

Cheers,

Martin

On 09/06/2015 11:05 AM, Hakeem Almabrazi wrote:
>
> Thank you Martin for your reply.
>
> 
>
> I will give it a try.  Can you please send me the name of the tool you
are referring to?
>
> 
>
> Also, I am hoping others will be able to share their insights about
the other concerns in my post.
>
> 
>
> Regards,
>
> 
>
> *From:*Martin Vickers [mjv08] [mailto:mj...@aber.ac.uk]
> *Sent:* Saturday, September 05, 2015 12:31 PM
> *To:* Hakeem Almabrazi
> *Cc:* galaxy-dev@lists.galaxyproject.org
> *Subject:* Re: [galaxy-dev] loading with Auto detect and fastqsanger
options issue!
>
> 
>
> Hi Hak,
>
> 
>
> I noticed the same thing when linking them when adding fastq.gz files
as a data library. The auto detect was reading the whole file as it was
unknown, presumably looking for some sort of characteristic to place it.
What I did to resolve this was add the archive data format data type
from the tool shed which included fastq.gz. This meant that auto detect
instantly found the correct type. Finally I added the gunzip tool to
galaxy and it works well.
>
> 
>
> There is also a trello feature request asking for this to be inbuilt
of if you're interested you may wish to vote for it. I will forward you
the like when I'm back at a computer.
>
> 
>
> Hope that helps,
>
> 
>
> Martin
>
> Sent on the move.
>
>
> On 4 Sep 2015, at 19:04, Hakeem Almabrazi <halmabr...@sidra.org
<mailto:halmabr...@sidra.org>> wrote:
>
> Hi,
>
> 
>
> I have installed Galaxy as cluster in my local env.  I have few
fastq.gz files in my file system that I want to link them to galaxy
instead of loading them.  When I select “Auto Detect” during the loading
I noticed two things.
>
> 
>
> First, it is taking way longer to link the files with the
AutoDetect option compared to when I chose Fastqsanger as my file type. 
It takes less than a second if I specify the type, i.e Fastqsanger.  
Any reason why is that?  Is there a way to speed up the first option
(Auto Detect)?
>
> 
>
> Second, when it finish linking for AutoDetect option, it shows the
DataType for these files as “data” rather than fastq files.
>
> Does this mean my fastq.gz files are not recognize as fastq files?
>
> 
>
> To validate my fastq files, I tried to use FastqValidator tool I
get an error saying “BGZF EOF marker is missing”.  How can I fix these
files?
>
> 
>
> I appreciate any kind of help,
>
> 
>
> Regards,
>
> Hak
>
> 
>
> 
>
> Disclaimer: This email and its attachments may be confidential and
are intended solely for the use of the individual to whom it is
addressed. If you are not the intended recipient, any reading, printing,
storage, disclosure, copying or any other action taken in respect of
this e-mail is prohibited and may be unlawful. If you are not the
intended recipient, please notify the sender immediately by using the
reply function and then permanently delete what you have received. Any
views or opinions expressed are solely those of the author and do not
necessarily represent those of Sidra Medical and Research Center.
>
> ___
> Please keep all replies on the list by using "reply all"
> in your mail client.  To manage your subscriptions to this
> and other Galaxy lists, please use the interface at:
>  https://lists.galaxyproject.org/
>
> To search Galaxy mailing lists use the unified search at:
>  http://galaxyproject.org/search/mailinglists/
>
> Disclaimer: This email and its attachments may be confidential and are
intended solely for the use of the individual to whom it is addressed.
If you are not the intended recipient, any reading, printing, storage,
disclosure, copying or any other action taken in respect of this e-mail
is prohibited and may be unlawful. If you are not the intended
recipient, please notify the sender immediately by using the reply
function and then permanently delete what you have received. Any views
or opinions expressed are solely those of the author and do not
necessarily represent those of Sidra Medical and Research Center.

- -- 

- --
Dr. Martin Vickers

Data Manager/HPC Systems Administrator
Institute of Biological, Environmental and Rural Sciences
IBERS N

Re: [galaxy-dev] loading with Auto detect and fastqsanger options issue!

2015-09-05 Thread Martin Vickers [mjv08]

Hi Hak,

I noticed the same thing when linking them when adding fastq.gz files as a data 
library. The auto detect was reading the whole file as it was unknown, 
presumably looking for some sort of characteristic to place it. What I did to 
resolve this was add the archive data format data type from the tool shed which 
included fastq.gz. This meant that auto detect instantly found the correct 
type. Finally I added the gunzip tool to galaxy and it works well.

There is also a trello feature request asking for this to be inbuilt of if 
you're interested you may wish to vote for it. I will forward you the like when 
I'm back at a computer.

Hope that helps,

Martin

Sent on the move.

On 4 Sep 2015, at 19:04, Hakeem Almabrazi 
> wrote:

Hi,

I have installed Galaxy as cluster in my local env.  I have few fastq.gz files 
in my file system that I want to link them to galaxy instead of loading them.  
When I select "Auto Detect" during the loading I noticed two things.

First, it is taking way longer to link the files with the AutoDetect option 
compared to when I chose Fastqsanger as my file type.  It takes less than a 
second if I specify the type, i.e Fastqsanger.   Any reason why is that?  Is 
there a way to speed up the first option (Auto Detect)?

Second, when it finish linking for AutoDetect option, it shows the DataType for 
these files as "data" rather than fastq files.
Does this mean my fastq.gz files are not recognize as fastq files?

To validate my fastq files, I tried to use FastqValidator tool I get an error 
saying "BGZF EOF marker is missing".  How can I fix these files?

I appreciate any kind of help,

Regards,
Hak


Disclaimer: This email and its attachments may be confidential and are intended 
solely for the use of the individual to whom it is addressed. If you are not 
the intended recipient, any reading, printing, storage, disclosure, copying or 
any other action taken in respect of this e-mail is prohibited and may be 
unlawful. If you are not the intended recipient, please notify the sender 
immediately by using the reply function and then permanently delete what you 
have received. Any views or opinions expressed are solely those of the author 
and do not necessarily represent those of Sidra Medical and Research Center.
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
 https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
 http://galaxyproject.org/search/mailinglists/
___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] LDAP Auto register - "username is None"

2015-09-03 Thread Martin Vickers

Hi Nicola,

So I've realised that none of that was actually the issue. The
(&(uid={username})(mail={email})) part does work fine, it's the setting
of the username that is the issue.  When the first unregistered user
logs in, it works fine but the username is set to -10. When a second
unregistered user attempts to login, they can't. If I manually change
their username, the second user is then able to log in and once again
the username is set to -10. (see attached images).

I think the issue here stems from;

{uid}

since I don't have a uid property in our ldap server. I've tried all
combinations of auto-register (True/False) and allow-register
(True/False/Challenge) and haven't been able to get it to work. It also
appears that auto-register-username and auto-register-email are
requirements to use this authenticator as without it noone can log in
(including registered users), and I get the following "Internal Server
Error" message.

This is my current auth_config.xml file;


ldap
True
ldaps://dc1.example.com
   
ou=Person,dc=dc1,dc=example,dc=com
   
cn=searchuser,ou=Person,dc=dc1,dc=example,dc=com
searchpasssword
uid,mail
   
((mail={email})(uid={username}))
False
{dn}
{password}
   
{uid}
{mail}



Doesn't one of the allow-register settings make/ask the user to provide
a username rather than trying to auto generate it? or, is there a way to
get the username out of the ldap server if it's not using uid to store it?

Many thanks,

Martin

On 09/02/2015 06:09 PM, Nicola Soranzo wrote:
> Hi Martin,
> I suspect there's an error in the sample auth_conf.xml file,
 should try to match only the email, not the username
(unless you specify True, in
which case it's viceversa) because it is not known when you first login.
In fact, for ActiveDirectory the filter is:
((objectClass=user)(mail={email}))
So, can you try to change:
((cn={username})(mail={email})) to
something like:
> (mail={email}) Cheers, Nicola
> On 02/09/15 15:51, Martin Vickers wrote:
>>
> Hi Nicola,
>
> It's an OpenLDAP server. uid isn't set on ours, it's cn instead, so
> using ldapsearch I can correctly bind;
>
> dn: cn=mjv08,ou=Person,dc=dc1,dc=example,dc=com
> objectClass: aberPerson
> cn: mjv08
>
> So authentication to the ldap server is working, the issue seems to be
> that when it's an unknown user, it's passing the following search string;
>
> (&(cn=None)(mail=unknownu...@aber.ac.uk))
>
> rather than;
>
> (&(cn=unknownuser)(mail=unknownu...@aber.ac.uk))
>
> hence the;
>
> galaxy.auth.providers.ldap_ad DEBUG 2015-09-02 15:40:07,322 LDAP
> authenticate: username is None
> galaxy.auth.providers.ldap_ad WARNING 2015-09-02 15:40:07,485 LDAP
> authenticate: search returned no results
>
> How is {username} in auth_config.xml set? Does it parse {email} to get it?
>
> Many thanks,
>
> Martin
>
> On 09/02/2015 03:38 PM, Nicola Soranzo wrote:
> > Hi Martin, > what LDAP server are you using? We have tested only
> OpenLDAP and > ActiveDirectory, but should work on any LDAP server. >
> > If it is OpenLDAP, I think you should use: > >
> uid,mail >
> ((mail={email})(uid={username})) >
> {uid} > > More
> details in: > >
> https://github.com/galaxyproject/galaxy/blob/dev/config/auth_conf.xml.sample
> > > Cheers, > Nicola > > Il 02.09.2015 15:03 Martin Vickers ha
> scritto: > > Hi All, > > I've been trying to get the new LDAP module
> to work. It works fine for > existing users but I can't get
> auto-register to work. In the logs I can > see the successful logins
> look like this; > > galaxy.webapps.galaxy.controllers.user DEBUG
> 2015-09-02 13:35:06,130 > trans.app.config.auth_config_file:
> ./config/auth_conf.xml > galaxy.auth.providers.ldap_ad DEBUG
> 2015-09-02 13:35:06,131 LDAP > authenticate: email is mj...@aber.ac.uk
> [1] > galaxy.auth.providers.ldap_ad DEBUG 2015-09-02 13:35:06,131 LDAP
> > authenticate: username is mjv08 >  >
> galaxy.auth.providers.ldap_ad DEBUG 2015-09-02 13:35:06,235 LDAP >
> authentication successful > > and those that are unsuccessful have a
> username as None, which is why > the search filter isn't working; > >
> galaxy.auth.providers.ldap_ad DEBUG 2015-09-02 13:47:13,951 LDAP >
> authenticate: email is unregu...@aber.ac.uk [2] >
> galaxy.auth.providers.ldap_ad DEBUG 2015-09-02 13:47:13,951 LDAP >
> authenticate: username is None >  > galaxy.auth.providers.ldap_ad
> WARN

Re: [galaxy-dev] LDAP Auto register - "username is None"

2015-09-03 Thread Martin Vickers


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi,

Okay I've solved it. The issue was that a) (mail={email}) is all that is
required in the search filter (to allow non-registered users) b) to
return cn and mail in the search-fields for use with the
auto-register-username and email. Finally, ensure auto-register is set
to True.

True
ldaps://dc1.example.com
ou=Person,dc=dc1,dc=example,dc=com
   
cn=searchuser,ou=Person,dc=dc1,dc=example,dc=com
searchpassword
cn,mail
(mail={email})
False
{dn}
{password}
{cn}
{mail}

Cheers,

Martin

On 09/03/2015 11:59 AM, Martin Vickers wrote:
> Hi Nicola,
>
> So I've realised that none of that was actually the issue. The
(&(uid={username})(mail={email})) part does work fine, it's the setting
of the username that is the issue.  When the first unregistered user
logs in, it works fine but the username is set to -10. When a second
unregistered user attempts to login, they can't. If I manually change
their username, the second user is then able to log in and once again
the username is set to -10. (see attached images).
>
> I think the issue here stems from;
>
> {uid}
>
> since I don't have a uid property in our ldap server. I've tried all
combinations of auto-register (True/False) and allow-register
(True/False/Challenge) and haven't been able to get it to work. It also
appears that auto-register-username and auto-register-email are
requirements to use this authenticator as without it noone can log in
(including registered users), and I get the following "Internal Server
Error" message.
>
> This is my current auth_config.xml file;
>
> 
> ldap
> True
> ldaps://dc1.example.com
>
ou=Person,dc=dc1,dc=example,dc=com
>
cn=searchuser,ou=Person,dc=dc1,dc=example,dc=com
> searchpasssword
> uid,mail
>
((mail={email})(uid={username}))
> False
> {dn}
> {password}
>
{uid}
> {mail}
> 
> 
>
> Doesn't one of the allow-register settings make/ask the user to
provide a username rather than trying to auto generate it? or, is there
a way to get the username out of the ldap server if it's not using uid
to store it?
>
> Many thanks,
>
> Martin
>
> On 09/02/2015 06:09 PM, Nicola Soranzo wrote:
> > Hi Martin,
> > I suspect there's an error in the sample auth_conf.xml file,
 should try to match only the email, not the username
(unless you specify True, in
which case it's viceversa) because it is not known when you first login.
In fact, for ActiveDirectory the filter is:
((objectClass=user)(mail={email}))
So, can you try to change:
((cn={username})(mail={email})) to
something like:
> > (mail={email}) Cheers, Nicola
> > On 02/09/15 15:51, Martin Vickers wrote:
> >>
>> Hi Nicola,
>>
>> It's an OpenLDAP server. uid isn't set on ours, it's cn instead, so
using ldapsearch I can correctly bind;
>>
>> dn: cn=mjv08,ou=Person,dc=dc1,dc=example,dc=com
>> objectClass: aberPerson
>> cn: mjv08
>>
>> So authentication to the ldap server is working, the issue seems to
be that when it's an unknown user, it's passing the following search string;
>>
>> (&(cn=None)(mail=unknownu...@aber.ac.uk))
>>
>> rather than;
>>
>> (&(cn=unknownuser)(mail=unknownu...@aber.ac.uk))
>>
>> hence the;
>>
>> galaxy.auth.providers.ldap_ad DEBUG 2015-09-02 15:40:07,322 LDAP
authenticate: username is None
>> galaxy.auth.providers.ldap_ad WARNING 2015-09-02 15:40:07,485 LDAP
authenticate: search returned no results
>>
>> How is {username} in auth_config.xml set? Does it parse {email} to
get it?
>>
>> Many thanks,
>>
>> Martin
>>
>> On 09/02/2015 03:38 PM, Nicola Soranzo wrote:
>> > Hi Martin, > what LDAP server are you using? We have tested only
OpenLDAP and > ActiveDirectory, but should work on any LDAP server. > >
If it is OpenLDAP, I think you should use: > >
uid,mail >
((mail={email})(uid={username})) >
{uid} > > More details
in: > >
https://github.com/galaxyproject/galaxy/blob/dev/config/auth_conf.xml.sample
> > Cheers, > Nicola > > Il 02.09.2015 15:03 Martin Vickers ha scritto:
> > Hi All, > > I've been trying to get the new LDAP module to work. It
works fine for > existing users but I can't get auto-register to work.
In the logs I can > see the successful logins loo

Re: [galaxy-dev] LDAP Auto register - "username is None"

2015-09-02 Thread Martin Vickers


-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Nicola,

It's an OpenLDAP server. uid isn't set on ours, it's cn instead, so
using ldapsearch I can correctly bind;

dn: cn=mjv08,ou=Person,dc=dc1,dc=example,dc=com
objectClass: aberPerson
cn: mjv08

So authentication to the ldap server is working, the issue seems to be
that when it's an unknown user, it's passing the following search string;

(&(cn=None)(mail=unknownu...@aber.ac.uk))

rather than;

(&(cn=unknownuser)(mail=unknownu...@aber.ac.uk))

hence the;

galaxy.auth.providers.ldap_ad DEBUG 2015-09-02 15:40:07,322 LDAP
authenticate: username is None
galaxy.auth.providers.ldap_ad WARNING 2015-09-02 15:40:07,485 LDAP
authenticate: search returned no results

How is {username} in auth_config.xml set? Does it parse {email} to get it?

Many thanks,

Martin

On 09/02/2015 03:38 PM, Nicola Soranzo wrote:
> Hi Martin,
> what LDAP server are you using? We have tested only OpenLDAP and
> ActiveDirectory, but should work on any LDAP server.
>
> If it is OpenLDAP, I think you should use:
>
> uid,mail
> ((mail={email})(uid={username}))
> {uid}
>
> More details in:
>
>
https://github.com/galaxyproject/galaxy/blob/dev/config/auth_conf.xml.sample
>
> Cheers,
> Nicola
>
> Il 02.09.2015 15:03 Martin Vickers ha scritto:
>
> Hi All,
>
> I've been trying to get the new LDAP module to work. It works fine for
> existing users but I can't get auto-register to work. In the logs I can
> see the successful logins look like this;
>
> galaxy.webapps.galaxy.controllers.user DEBUG 2015-09-02 13:35:06,130
> trans.app.config.auth_config_file: ./config/auth_conf.xml
> galaxy.auth.providers.ldap_ad DEBUG 2015-09-02 13:35:06,131 LDAP
> authenticate: email is mj...@aber.ac.uk [1]
> galaxy.auth.providers.ldap_ad DEBUG 2015-09-02 13:35:06,131 LDAP
> authenticate: username is mjv08
> 
> galaxy.auth.providers.ldap_ad DEBUG 2015-09-02 13:35:06,235 LDAP
> authentication successful
>
> and those that are unsuccessful have a username as None, which is why
> the search filter isn't working;
>
> galaxy.auth.providers.ldap_ad DEBUG 2015-09-02 13:47:13,951 LDAP
> authenticate: email is unregu...@aber.ac.uk [2]
> galaxy.auth.providers.ldap_ad DEBUG 2015-09-02 13:47:13,951 LDAP
> authenticate: username is None
> 
> galaxy.auth.providers.ldap_ad WARNING 2015-09-02 13:47:14,110 LDAP
> authenticate: search returned no results
>
> My auth_config.xml openldap authenticator looks like this (edited to
> remove openldap server details);
>
> ldap
> '{email}'.endswith('@example.com')
>
> True
> Challenge
> ldaps://dc1.example.com
>
> ou=People,dc=dc1,dc=example,dc=com
>
> cn=searchuser,ou=People,dc=dc1,dc=example,dc=com
>
> searchuserpassword
> cn,mail
>
> (&(cn={username})(mail={email}))
> {dn}
> {password}
>
> {cn}
> {mail}
>
> Are there any settings in galaxy.ini that are required to enable this to
> work?
>
> Many thanks
>
> Martin
>
>
>
> Connetti gratis il mondo con la nuova indoona:  hai la chat, le
chiamate, le video chiamate e persino le chiamate di gruppo.
> E chiami gratis anche i numeri fissi e mobili nel mondo!
> Scarica subito l’app Vai su https://www.indoona.com/
>
>

- -- 

- --
Dr. Martin Vickers

Data Manager/HPC Systems Administrator
Institute of Biological, Environmental and Rural Sciences
IBERS New Building
Aberystwyth University

w: http://www.martin-vickers.co.uk/
e: mj...@aber.ac.uk
t: 01970 62 2807
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.14 (GNU/Linux)

iQEcBAEBAgAGBQJV5wzhAAoJEHa0a8GkKQgIdGIH/3yjT7hz+3IECPIak4qyiEbF
C/4s+gpQdKnQHMJrg0xB1aB7lXhO+LjgP9bkZLMwBlQpiOPz2cApZ9e51S+vIXEU
e+MoOYIXputDgG49pfl6TB9N0fR2FIZcnp5vy3GBFUIWreJRvRX2EuiI97iY7iei
eSg9cjZ6UIWZBKdo+PrO1hPdhkAX+l5Kd8HMipLuInKpvZDZfiBxQMd4zFCIGz3W
vSymyQSHQpOul3rnwp70l76doT9jqsBW3ggpnwdbP2/pgRLvmPkyvCh2u2fyrouv
vsj11ODrskIZb10YyXy5QxsbluaThA1QeTw+0s+UEIPrNvyLcrSmuidHDjlnV5I=
=zSFZ
-END PGP SIGNATURE-

___
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

[galaxy-dev] ./run.sh segfault

2015-07-16 Thread Martin Vickers

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi All,

I have a weird issue that's just cropped up. After a new install of
galaxy (checked out on Monday from github) on a ubuntu vm, using
postgres rather than sqlite as well as a few other production
recommendations, I started playing around with the Data Libraries
functionality. I linked a bunch of fastq.gz files into galaxy (around
150 in total) and everything was working fine. I went home and the
next day, it was down.

I tried to start it up as usual (using an init.d script), it worked
for less than a minute and then disappeared again. So I tried running
it as the galaxy user using ./run.sh and I get a seg fault;

Starting server in PID 23173.
serving on http://144.124.110.39:8080
Segmentation fault

Tried again with strace

Starting server in PID 23552.
serving on http://144.124.110.39:8080
[{WIFSIGNALED(s)  WTERMSIG(s) == SIGKILL}], 0, NULL) = 23552
- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=23552,
si_status=SIGKILL, si_utime=1590, si_stime=1930} ---
rt_sigreturn()  = 23552
write(2, Killed\n, 7Killed
) = 7
read(10, , 8192)  = 0
exit_group(137) = ?
+++ exited with 137 +++

I can't see anything odd in the log file and I've turned debugging on
in galaxy.ini. I'm at a bit of a loss. Does anyone know what might be
causing it?

Cheers,

- -- 
Dr. Martin Vickers

Data Manager/HPC Systems Administrator
Institute of Biological, Environmental and Rural Sciences
IBERS New Building
Aberystwyth University

w: http://www.martin-vickers.co.uk/
e: mj...@aber.ac.uk
t: 01970 62 2807
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.14 (GNU/Linux)

iQEcBAEBAgAGBQJVp80YAAoJEHa0a8GkKQgIGlIH/1VfAPbs/5ApDBdyoOV5qf1y
oCOv93IojARyfI0ksSjF8NRNzw5fNp1R8AzZzomaR3SOUkBuZutre600sy0azTZw
E6gjxtMuvaMyEsOTXtToVarVJT0wTG8+5DJRIYLxtYZm7kvbZK0WuzrN2zDT6663
Rnm7zI/zBpTAyp6uXwgmz0x5gpH6KFwRcEHEbU3JWy6nj1zithJShwYPlBuhT5IB
OaPwOKflcZpZ8NBTEGsh038JrkU+eE50a9aEjQ2m/DpfM/TN9ujgEFm1dyy/iQS7
ewwQUpWJDkA/u0ZX602dsNdV2LvGuKVVMEHiQ25zaUQZ/iGTwKBQsFM2LlDybgA=
=jzYG
-END PGP SIGNATURE-
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Re: [galaxy-dev] ./run.sh segfault

2015-07-16 Thread Martin Vickers [mjv08]

Hi Nate,

Thanks for the reply. In syslog I'm getting;


Jul 16 20:36:41 galaxy kernel: [  117.123921] Out of memory: Kill process 1390 
(python) score 986 or sacrifice child
Jul 16 20:36:41 galaxy kernel: [  117.124087] Killed process 1390 (python) 
total-vm:43496348kB, anon-rss:32611892kB, file-rss:1800kB
(END)

It's a 32GB VM. I could increase it but I wouldn't expect 32GB to be too 
little. I've attached the full syslog.


Dr. Martin Vickers

Data Manager/HPC Systems Administrator
Institute of Biological, Environmental and Rural Sciences
IBERS New Building
Aberystwyth University

w: http://www.martin-vickers.co.uk/
e: mj...@aber.ac.uk
t: 01970 62 2807



From: Nate Coraor n...@bx.psu.edu
Sent: 16 July 2015 04:36 PM
To: Martin Vickers [mjv08]
Cc: galaxy-dev@lists.galaxyproject.org
Subject: Re: [galaxy-dev] ./run.sh segfault

Hi Martin,

Is there anything in the syslog?

--nate

On Thu, Jul 16, 2015 at 11:26 AM, Martin Vickers 
mj...@aber.ac.ukmailto:mj...@aber.ac.uk wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi All,

I have a weird issue that's just cropped up. After a new install of
galaxy (checked out on Monday from github) on a ubuntu vm, using
postgres rather than sqlite as well as a few other production
recommendations, I started playing around with the Data Libraries
functionality. I linked a bunch of fastq.gz files into galaxy (around
150 in total) and everything was working fine. I went home and the
next day, it was down.

I tried to start it up as usual (using an init.d script), it worked
for less than a minute and then disappeared again. So I tried running
it as the galaxy user using ./run.sh and I get a seg fault;

Starting server in PID 23173.
serving on http://144.124.110.39:8080
Segmentation fault

Tried again with strace

Starting server in PID 23552.
serving on http://144.124.110.39:8080
[{WIFSIGNALED(s)  WTERMSIG(s) == SIGKILL}], 0, NULL) = 23552
- --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=23552,
si_status=SIGKILL, si_utime=1590, si_stime=1930} ---
rt_sigreturn()  = 23552
write(2, Killed\n, 7Killed
) = 7
read(10, , 8192)  = 0
exit_group(137) = ?
+++ exited with 137 +++

I can't see anything odd in the log file and I've turned debugging on
in galaxy.ini. I'm at a bit of a loss. Does anyone know what might be
causing it?

Cheers,

- --
Dr. Martin Vickers

Data Manager/HPC Systems Administrator
Institute of Biological, Environmental and Rural Sciences
IBERS New Building
Aberystwyth University

w: http://www.martin-vickers.co.uk/
e: mj...@aber.ac.ukmailto:mj...@aber.ac.uk
t: 01970 62 2807
-BEGIN PGP SIGNATURE-
Version: GnuPG v2.0.14 (GNU/Linux)

iQEcBAEBAgAGBQJVp80YAAoJEHa0a8GkKQgIGlIH/1VfAPbs/5ApDBdyoOV5qf1y
oCOv93IojARyfI0ksSjF8NRNzw5fNp1R8AzZzomaR3SOUkBuZutre600sy0azTZw
E6gjxtMuvaMyEsOTXtToVarVJT0wTG8+5DJRIYLxtYZm7kvbZK0WuzrN2zDT6663
Rnm7zI/zBpTAyp6uXwgmz0x5gpH6KFwRcEHEbU3JWy6nj1zithJShwYPlBuhT5IB
OaPwOKflcZpZ8NBTEGsh038JrkU+eE50a9aEjQ2m/DpfM/TN9ujgEFm1dyy/iQS7
ewwQUpWJDkA/u0ZX602dsNdV2LvGuKVVMEHiQ25zaUQZ/iGTwKBQsFM2LlDybgA=
=jzYG
-END PGP SIGNATURE-
___
Please keep all replies on the list by using reply all
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Jul 16 20:17:01 galaxy CRON[2435]: (root) CMD (   cd /  run-parts --report 
/etc/cron.hourly)
Jul 16 20:22:10 galaxy puppet-agent[2643]: Finished catalog run in 0.25 seconds
Jul 16 20:32:37 galaxy kernel: [121243.785394] init: tty4 main process (994) 
killed by TERM signal
Jul 16 20:32:37 galaxy kernel: [121243.785744] init: tty5 main process (997) 
killed by TERM signal
Jul 16 20:32:37 galaxy kernel: [121243.786083] init: tty2 main process (1003) 
killed by TERM signal
Jul 16 20:32:37 galaxy kernel: [121243.786428] init: tty3 main process (1004) 
killed by TERM signal
Jul 16 20:32:37 galaxy kernel: [121243.786787] init: tty6 main process (1006) 
killed by TERM signal
Jul 16 20:32:37 galaxy kernel: [121243.787663] init: cron main process (1036) 
killed by TERM signal
Jul 16 20:32:37 galaxy kernel: [121243.788318] init: irqbalance main process 
(1069) killed by TERM signal
Jul 16 20:32:37 galaxy kernel: [121243.788853] init: hvc0 main process (1806) 
killed by TERM signal
Jul 16 20:32:37 galaxy kernel: [121243.790003] init: tty1 main process (1810) 
killed by TERM signal
Jul 16 20:32:37 galaxy kernel: [121243.790710] init: plymouth-upstart-bridge 
main process (3302) terminated with status 1
Jul 16 20:32:37 galaxy kernel: [121243.790729] init: plymouth-upstart-bridge 
main process ended, respawning
Jul 16 20:32:37 galaxy rsyslogd: [origin software=rsyslogd swVersion=7.4.4 
x-pid=744 x-info=http://www.rsyslog.com;] exiting

Re: [galaxy-dev] Creating R galaxy wrapper where a function checks file extension

[galaxy-dev] Creating R galaxy wrapper where a function checks file extension

Re: [galaxy-dev] Creating R galaxy wrapper where a function checks file extension

Re: [galaxy-dev] Adding data libraries from filesystem path creating duplicates

Re: [galaxy-dev] loading with Auto detect and fastqsanger options issue!

Re: [galaxy-dev] loading with Auto detect and fastqsanger options issue!

Re: [galaxy-dev] LDAP Auto register - "username is None"

Re: [galaxy-dev] LDAP Auto register - "username is None"

Re: [galaxy-dev] LDAP Auto register - "username is None"

[galaxy-dev] ./run.sh segfault

Re: [galaxy-dev] ./run.sh segfault

11 matches

Site Navigation

Mail list logo

Footer information