-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi John,

Thanks for taking the time to reply. I never thought to look at the
proxy settings but I think you're right, the behaviour seems to match
what you've described.

Like you I'm not really an expert on proxies and have no idea what would
be mis-configured that would cause this.

I'm using nginx and the configuration is as described in the wiki. I've
not loaded any special extensions.

nginx is configured like this;

upstream galaxy_app {
    server localhost:8090;
    server localhost:8091;
    server localhost:8092;
    server localhost:8093;
    server localhost:8094;
    server localhost:8095;
}

server {
    # pass to uWSGI by default
    location / {
        proxy_pass http://galaxy_app;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header X-URL-SCHEME https;
    }

....static content

}

and in galaxy.ini I have a bunch of handlers, e.g.


[server:handler0]
use = egg:Paste#http
port = 8090
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

I thought that maybe the issue was to do with the 'job admin
complication' very briefly mentioned here;

https://production-galaxy-instances-with-cloudman-and-cloudbiolinux.readthedocs.org/en/latest/

so I added this to my nginx conf

    location /admin/jobs {
        proxy_pass  http://localhost:8090;
    }


so this complication is not the one I'm having here.

Are any of the people John mentioned having this issue here on the dev
board?

Cheers,

Martin

On 09/14/2015 03:11 PM, John Chilton wrote:
> If I had to guess, I would guess this is caused by a mis-configured
> proxy (nginx or Apache) that is resubmitting a POST request that is
> taking Galaxy to long to respond to. Order of events being something
> like:
>
> - User clicks to upload library items.
> - Proxy gets requests and passes to Galaxy
> - Galaxy takes a long time to process request and doesn't respond
> within a timeout.
> - Proxy resends POST request to Galaxy.
> - Galaxy takes a long time to process request and doesn't respond
> within a timeout.
> ...
>
> Proxies should never resend POST requests to Galaxy as far a I can
> imagine, but we have seen this for instance when submitting workflows.
> Some people have had their proxy retry that request repeatedly.
>
> I don't really know if this is a problem with the default proxy
> configurations we list on the wiki or if it comes down to
> customizations or special loaded extensions at various sites that have
> encountered this.
>
> Is this enough to help debug the problem? I'm not really an expert on
> specific proxies, etc... and you have it there and seem to be able to
> reproduce the problem. If you do want further help I would post the
> proxy you are using, the extensions, the configuration, and the Galaxy
> logs corresponding to this incident to see if we can see the repeated
> posts and the route that is being posted to.
>
> If you are not using a proxy, then I am stumped :(.
>
> -John
>
>
> On Fri, Sep 4, 2015 at 12:04 PM, Martin Vickers <mj...@aber.ac.uk> wrote:
>> Hi All,
>>
>> I've noticed an issue a couple of times now where I've added a directory
>> of fastq's from an NFS mounted filesystem (reference only rather than
>> copying into galaxy) and then galaxy times out. Load average begins to
>> get really high and then consumes all the RAM and sometimes crashes.
>> These are the same symptom as I had before with this issue that was
>> never resolved;
>>
>> http://dev.list.galaxyproject.org/run-sh-segfault-td4667549.html#a4667553
>>
>> What I've noticed is that in the dataset I'm uploading to galaxy, there
>> are suddenly many duplicates. In this example that's just happened,
>> there are 288 fastq.gz files in the physical folder, but galaxy has
>> created 6 references to each file resulting in 1728 datasets in the
>> folder (see attached images).
>>
>> When this happened before and crashed the galaxy application, whenever
>> it restarted it'd try to resume what it was doing which created an
>> endless loop of retrying and crashing until the job was removed.
>>
>> Does anyone know what may be causing this?
>>
>> Cheers,
>>
>> Martin
>>
>> --
>>
>> --
>> Dr. Martin Vickers
>>
>> Data Manager/HPC Systems Administrator
>> Institute of Biological, Environmental and Rural Sciences
>> IBERS New Building
>> Aberystwyth University
>>
>> w: http://www.martin-vickers.co.uk/
>> e: mj...@aber.ac.uk
>> t: 01970 62 2807
>>
>>
>> ___________________________________________________________
>> Please keep all replies on the list by using "reply all"
>> in your mail client.  To manage your subscriptions to this
>> and other Galaxy lists, please use the interface at:
>>   https://lists.galaxyproject.org/
>>
>> To search Galaxy mailing lists use the unified search at:
>>   http://galaxyproject.org/search/mailinglists/

- -- 

- --
Dr. Martin Vickers

Data Manager/HPC Systems Administrator
Institute of Biological, Environmental and Rural Sciences
IBERS New Building
Aberystwyth University
SY23 3FG

w: http://www.martin-vickers.co.uk/
e: mj...@aber.ac.uk
t: 01970 62 2807
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)

iQEcBAEBAgAGBQJV/81wAAoJEHa0a8GkKQgIKD8IAIMP5gYGAjX62u129JboJtWL
chSzRp3GN8ZGC5qosueMwlcNFdfTObDeM81z7vcXz1FT1N0mBXSMFEv1YaCc7ZrW
YEKIc9m6n2qR4dqEwRXlXLfDv6A7Unoq+T5DG4s+teb0RfIg457KT2ilu9PVjeS0
408lAlIvwiutK2msa1/RsiaYwtXKVMPVAXNukP+KN5StYTkNt7rDDGoy7fwnx+FP
P/JXTP9jrHRLNl+Uegx/oaNCm/R6WV9h+vRx1xXFe45M9RfHZSxF0fbGvqMDOPg+
vzvbClZwuxNuafFTB0E5NnqppN13MLt8Sc2wDT1PFeXAUqR2N4ZOQwM5yywXWAs=
=gilj
-----END PGP SIGNATURE-----


___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to