Thanks – I’ve attached the log. I just tried to start a worker and let it go to 
the first reboot and then copied this log.  I logged into the worker and it 
looks ok (dmesg, etc) the only noticable thing was /mnt is empty (just a 
lost+found directory) and I was expecting to see an nfs mount for galaxy export 
or something.  But I’m still finding my way round the system.  It may also have 
been the time in the reboot cycle that I was there.

Best wishes,

Nick
--
Nick Dickens
DPhil BSc ARCS

Bioinformatics Team Leader
Wellcome Trust Centre for Molecular Parasitology
B6-21 SGDB
120 University Place
Glasgow
G12 8TA

Tel: +44 141 330 8282

http://fb.me/WTCMPbix
@WTCMPbix
http://www.gla.ac.uk/researchinstitutes/iii/staff/nickdickens/
http://www.gla.ac.uk/researchinstitutes/iii/staff/jeremymottram/comparativegenomicsofleishmania/

From: Enis Afgan <enis.af...@irb.hr<mailto:enis.af...@irb.hr>>
Date: Friday, 19 June 2015 17:00
To: Nick Dickens <nick.dick...@glasgow.ac.uk<mailto:nick.dick...@glasgow.ac.uk>>
Cc: 
"galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>" 
<galaxy-dev@lists.galaxyproject.org<mailto:galaxy-dev@lists.galaxyproject.org>>
Subject: Re: [spam?] [galaxy-dev] Cloudman cluster not starting workers

Hi Nick,
Sorry to hear you're having trouble. I just tried a couple of scenarios and 
they all worked as expected (e.g., with and without elastic IPs, different 
instance types).

The main CloudMan log is located in /mnt/cm/paster.log, on both master and 
worker instances (if you didn't download the ssh key from cloudlaunch, you can 
ssh with ubuntu username and the same password as provided on the cloudlaunch 
form). The log is also available from the UI if you go to Admin page and then 
click 'Show CloudMan log' under 'System controls'. If you can share that, we 
can hopefully figure out what's going on.

Best,
Enis

On Wed, Jun 17, 2015 at 11:47 AM, Nicholas Dickens 
<nick.dick...@glasgow.ac.uk<mailto:nick.dick...@glasgow.ac.uk>> wrote:
Hi All,

First time post, as a quick intro, I’m have some reasonable experience with EC2 
& AWS for developing our own pipelines, I’m comfortable in python and *nix 
flavours and I am developing a completely custom galaxy in AWS for members of 
the WTCMP, Glasgow.  In the meantime, I have thrown up a quick cloudman galaxy 
using the cloudstart and the cloudman 2.3 ami (ami-a7dbf6ce) in us-east-1.  
Auto–scaling didn’t seem to work so I’ve switched it off and added nodes 
manually, I tried various sizes including ‘same as master’ instances but they 
just don’t start – in the EC2 console I can see them and see them running.  But 
they’re constantly pending in the /cloud interface and in the log they reboot 4 
times and then terminate – apparently not responding "10:16:56 - Instance 
i-xxxxxx not responding after 4 reboots. Terminating instance".

It’s out of the box, I editted the universe_wsgi.ini… file to disallow user 
registration and allow me to impersonate users but didn’t change anything else. 
 The only other configuration I’ve done is associate an elastic IP with the 
master instance so I can have a more static url for a couple of test users (if 
I need to destroy it and start again, etc).

I’m new to the system so don’t know which logs are best to check…and am I 
missing something obvious?  It there a known bug when using elastic IPs? I’ve 
googled but with no joy.

Thanks for your help and best wishes,

Nick
--
Nick Dickens
DPhil BSc ARCS

Bioinformatics Team Leader
Wellcome Trust Centre for Molecular Parasitology
B6-21 SGDB
120 University Place
Glasgow
G12 8TA

Tel: +44 141 330 8282<tel:%2B44%20141%20330%208282>

http://fb.me/WTCMPbix
@WTCMPbix
http://www.gla.ac.uk/researchinstitutes/iii/staff/nickdickens/
http://www.gla.ac.uk/researchinstitutes/iii/staff/jeremymottram/comparativegenomicsofleishmania/


----------------------------- Upozorenje -----------------------------

Automatskom detekcijom utvrdjeno je da tekst ove poruke
podsjeca na tzv. phishing poruku.

AKO SE U PORUCI TRAZI DA POSALJETE VASU IRB LOZINKU ILI
DA UNESETE IRB PODATKE NA NAVEDENOM LINKU, RADI SE O
NAPADU S CILJEM KRADJE I ZLOUPOTREBE PODATAKA.

Centar za informatiku i racunarstvo,
Institut Rudjer Boskovic

----------------------------- Upozorenje -----------------------------

___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

        
CloudMan from Galaxy
Admin | Report bugs | Wiki | Screencast  
Only up to the most recent 200 lines of the file (paster.log) are shown. Show 
all | Show less | Show more | Back to admin view
2015-06-19 20:42:32,573 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:43:36,824 DEBUG            sge:552  qstat: 
['al...@ip-172-31-2-194.ec2.inte BIP   0/0/2          0.13     lx24-amd64    ']
2015-06-19 20:43:37,425 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:44:41,676 DEBUG            sge:552  qstat: 
['al...@ip-172-31-2-194.ec2.inte BIP   0/0/2          0.16     lx24-amd64    ']
2015-06-19 20:44:42,432 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:45:42,692 DEBUG            sge:552  qstat: 
['al...@ip-172-31-2-194.ec2.inte BIP   0/0/2          0.12     lx24-amd64    ']
2015-06-19 20:45:43,248 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:46:47,500 DEBUG            sge:552  qstat: 
['al...@ip-172-31-2-194.ec2.inte BIP   0/0/2          0.11     lx24-amd64    ']
2015-06-19 20:46:48,223 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:47:01,584 DEBUG            sge:498  Removing instance 
'i-fd327802' with FQDN '172.31.2.194' from SGE execution host list (including 
@allhosts)
2015-06-19 20:47:01,585 DEBUG            sge:414  to_add: 'None'
2015-06-19 20:47:01,585 DEBUG            sge:415  to_remove: '172.31.2.194'
2015-06-19 20:47:01,585 DEBUG            sge:418  Composing SGE's @allhosts 
group config file /tmp/ah_remove_20_47_01:
2015-06-19 20:47:01,585 DEBUG            sge:425   - master is marked as 
non-exec host and will not be included in @allhosts file
2015-06-19 20:47:01,585 DEBUG            sge:454  new_allhosts:
group_name @allhosts 
hostlist NONE

2015-06-19 20:47:01,585 DEBUG            sge:456  New SGE @allhosts file 
written successfully to /tmp/ah_remove_20_47_01.
2015-06-19 20:47:01,736 DEBUG            sge:507  Successfully updated 
@allhosts to remove 'i-fd327802'
2015-06-19 20:47:02,051 DEBUG            sge:526  Successfully removed instance 
'i-fd327802' with IP '172.31.2.194' from SGE execution host list.
2015-06-19 20:47:02,052 INFO          master:845  The master instance has been 
set to *not* execute jobs.  To manually change this, use the cloudman admin 
panel.
2015-06-19 20:47:02,052 INFO             ec2:443  Adding 1 on-demand instance(s)
2015-06-19 20:47:02,054 DEBUG            ec2:474  Starting instance(s) in VPC 
with the following command : ec2_conn.run_instances( image_id='ami-a7dbf6ce', 
min_count='1', max_count='1', key_name='cloudman_key_pair', 
security_group_ids=['sg-00858065'], user_data(with password/secret_key filtered 
out)=[s3_port: None
cloud_type: ec2
master_public_ip: 54.88.49.203
freenxpass: !galaxySATDEVZGK
cluster_type: Galaxy
tags: {u'i-b8083047': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 
'Worker: icam_galaxy'}, u'i-58edd5a7': {'clusterName': u'icam_galaxy', 'role': 
'worker', 'Name': 'Worker: icam_galaxy'}, u'i-9ff2cf60': {'clusterName': 
u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, 
u'i-fd327802': {'clusterName': u'icam_galaxy', 'role': 'master', 'Name': 
'master: icam_galaxy'}, u'i-e9251816': {'clusterName': u'icam_galaxy', 'role': 
'worker', 'Name': 'Worker: icam_galaxy'}, u'i-98f2cf67': {'clusterName': 
u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, 
u'i-e21b261d': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 
'Worker: icam_galaxy'}, u'i-210c0ade': {'clusterName': u'icam_galaxy', 'role': 
'worker', 'Name': 'Worker: icam_galaxy'}, u'i-2edfe4d1': {'clusterName': 
u'icam_galaxy', 'role': 'worker', 'Name': 'Worker: icam_galaxy'}, 
u'i-64edd59b': {'clusterName': u'icam_galaxy', 'role': 'worker', 'Name': 
'Worker: icam_galaxy'}, u'i-b9083046': {'clusterName': u'icam_galaxy', 'role': 
'worker', 'Name': 'Worker: icam_galaxy'}, u'vol-86b205c0': {'clusterName': 
u'icam_galaxy'}, u'i-e31b261c': {'clusterName': u'icam_galaxy', 'role': 
'worker', 'Name': 'Worker: icam_galaxy'}, u'vol-d7bd0a91': {'clusterName': 
u'icam_galaxy', 'Name': 'galaxyIndicesFS', 'bucketName': 
'cm-442caa28f69846327455b14f155f75cd', 'roles': 'galaxyIndices', 'filesystem': 
'galaxyIndices'}, u'vol-8ebd0ac8': {'clusterName': u'icam_galaxy', 'Name': 
'galaxyFS', 'bucketName': 'cm-442caa28f69846327455b14f155f75cd', 'roles': 
'galaxyTools,galaxyData', 'filesystem': 'galaxy'}}
master_ip: 172.31.2.194
cluster_name: icam_galaxy
persistent_data_version: 3
machine_image_id: ami-a7dbf6ce
role: worker
bucket_cluster: cm-442caa28f69846327455b14f155f75cd
boot_script_path: /tmp/cm
master_hostname: ip-172-31-2-194.ec2.internal
ec2_conn_path: /
region_name: us-east-1
region_endpoint: ec2.amazonaws.com
ec2_port: None
deployment_version: 2
filesystems: [{'kind': 'volume', 'mount_point': '/mnt/galaxy', 'name': 
'galaxy', 'roles': ['galaxyTools', 'galaxyData'], 'ids': [u'vol-8ebd0ac8']}, 
{'kind': 'snapshot', 'mount_point': '/mnt/galaxyIndices', 'name': 
'galaxyIndices', 'roles': ['galaxyIndices'], 'ids': ['snap-4b20f451']}]
cloud_name: Amazon
boot_script_name: cm_boot.py
services: [{'name': 'Migration', 'roles': ['Migration']}, {'name': 'SGE', 
'roles': ['SGE']}, {'name': 'Postgres', 'roles': ['Postgres']}, {'name': 
'ProFTPd', 'roles': ['ProFTPd']}, {'home': '/mnt/galaxy/galaxy-app', 'name': 
'Galaxy', 'roles': ['Galaxy']}, {'name': 'GalaxyReports', 'roles': 
['GalaxyReports']}]
is_secure: True
access_key: AKIAJQJYPZIMLKSDWHAQ
placement: us-east-1a
cloudman_home: /mnt/cm
bucket_default: cloudman
s3_host: s3.amazonaws.com
s3_conn_path: /], instance_type='c3.large', placement='us-east-1a', 
subnet_id='subnet-e19863b8')
2015-06-19 20:47:05,720 DEBUG            ec2:409  Adding tag 
'clusterName:icam_galaxy' to resource 'i-69a8a496'
2015-06-19 20:47:05,908 DEBUG            ec2:409  Adding tag 'role:worker' to 
resource 'i-69a8a496'
2015-06-19 20:47:06,056 DEBUG            ec2:409  Adding tag 'Name:Worker: 
icam_galaxy' to resource 'i-69a8a496'
2015-06-19 20:47:06,249 DEBUG            ec2:517  Adding Instance 
Instance:i-69a8a496
2015-06-19 20:47:06,250 DEBUG            ec2:531  Started 1 instance(s)
2015-06-19 20:47:08,462 DEBUG            sge:552  qstat: []
2015-06-19 20:47:09,121 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:47:09,380 DEBUG         master:3024 Got public IP for instance 
i-69a8a496: 54.164.165.200
2015-06-19 20:47:09,380 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 3 secs ago); will wait a 
bit longer before a check...
2015-06-19 20:47:21,618 DEBUG            sge:552  qstat: []
2015-06-19 20:47:22,372 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:47:22,373 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 16 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:47:34,610 DEBUG            sge:552  qstat: []
2015-06-19 20:47:35,346 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:47:35,347 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 29 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:47:47,587 DEBUG            sge:552  qstat: []
2015-06-19 20:47:48,149 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:47:48,150 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:47:48,251 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:47:48,251 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:47:48 
| last m_state change before 0:00:42 | time_rebooted before 20:47:48
2015-06-19 20:48:00,497 DEBUG            sge:552  qstat: []
2015-06-19 20:48:01,104 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:48:01,104 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 12 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:48:13,341 DEBUG            sge:552  qstat: []
2015-06-19 20:48:13,872 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:48:13,873 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 25 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:48:26,112 DEBUG            sge:552  qstat: []
2015-06-19 20:48:26,654 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:48:26,655 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:48:26,721 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:48:26,721 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:48:26 
| last m_state change before 0:01:20 | time_rebooted before 20:48:26
2015-06-19 20:48:38,962 DEBUG            sge:552  qstat: []
2015-06-19 20:48:39,578 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:48:39,578 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 12 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:48:51,819 DEBUG            sge:552  qstat: []
2015-06-19 20:48:52,390 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:48:52,391 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 25 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:49:04,626 DEBUG            sge:552  qstat: []
2015-06-19 20:49:05,181 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:49:05,181 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:49:05,283 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:49:05,283 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:49:05 
| last m_state change before 0:01:59 | time_rebooted before 20:49:05
2015-06-19 20:49:17,520 DEBUG            sge:552  qstat: []
2015-06-19 20:49:18,146 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:49:18,147 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 12 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:49:30,384 DEBUG            sge:552  qstat: []
2015-06-19 20:49:31,572 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:49:31,572 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 26 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:49:43,809 DEBUG            sge:552  qstat: []
2015-06-19 20:49:44,426 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:49:44,427 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:49:44,502 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:49:44,502 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:49:44 
| last m_state change before 0:02:38 | time_rebooted before 20:49:44
2015-06-19 20:49:56,743 DEBUG            sge:552  qstat: []
2015-06-19 20:49:57,272 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:49:57,273 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 12 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:50:09,520 DEBUG            sge:552  qstat: []
2015-06-19 20:50:10,102 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:50:10,103 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 25 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:50:22,342 DEBUG            sge:552  qstat: []
2015-06-19 20:50:22,982 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:50:22,983 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:50:23,058 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:50:23,058 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:50:23 
| last m_state change before 0:03:16 | time_rebooted before 20:50:23
2015-06-19 20:50:35,300 DEBUG            sge:552  qstat: []
2015-06-19 20:50:35,857 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:50:35,858 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 12 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:50:48,163 DEBUG            sge:552  qstat: []
2015-06-19 20:50:48,751 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:50:48,752 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 25 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:51:00,990 DEBUG            sge:552  qstat: []
2015-06-19 20:51:01,652 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:51:01,653 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:51:01,758 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:51:01,758 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:51:01 
| last m_state change before 0:03:55 | time_rebooted before 20:51:01
2015-06-19 20:51:13,997 DEBUG            sge:552  qstat: []
2015-06-19 20:51:14,869 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:51:14,870 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 13 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:51:27,125 DEBUG            sge:552  qstat: []
2015-06-19 20:51:27,787 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:51:27,788 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 26 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:51:40,053 DEBUG            sge:552  qstat: []
2015-06-19 20:51:40,577 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:51:40,578 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:51:40,644 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:51:40,645 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:51:40 
| last m_state change before 0:04:34 | time_rebooted before 20:51:40
2015-06-19 20:51:52,884 DEBUG            sge:552  qstat: []
2015-06-19 20:51:53,403 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:51:53,404 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 12 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:52:05,641 DEBUG            sge:552  qstat: []
2015-06-19 20:52:06,206 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:52:06,207 DEBUG         master:2562 Instance 'i-69a8a496' (IP: 
54.164.165.200) has been quiet for a while (last check 25 secs ago); will wait 
a bit longer before a check...
2015-06-19 20:52:38,452 DEBUG            sge:552  qstat: []
2015-06-19 20:52:38,996 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:52:38,996 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:52:39,093 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:52:39,093 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:52:39 
| last m_state change before 0:05:32 | time_rebooted before 20:52:39
2015-06-19 20:53:11,370 DEBUG            sge:552  qstat: []
2015-06-19 20:53:12,115 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:53:12,115 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:53:12,217 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:53:12,217 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:53:12 
| last m_state change before 0:06:05 | time_rebooted before 20:53:12
2015-06-19 20:53:44,462 DEBUG            sge:552  qstat: []
2015-06-19 20:53:45,254 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:53:45,254 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:53:45,348 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:53:45,348 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:53:45 
| last m_state change before 0:06:39 | time_rebooted before 20:53:45
2015-06-19 20:54:17,595 DEBUG            sge:552  qstat: []
2015-06-19 20:54:18,145 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:54:18,146 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:54:18,223 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:54:18,223 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:54:18 
| last m_state change before 0:07:11 | time_rebooted before 20:54:18
2015-06-19 20:54:18,223 INFO          master:2810 Rebooting instance 
'i-69a8a496' (IP: 54.164.165.200) (reboot #1).
2015-06-19 20:54:18,331 DEBUG         master:2823 Incremented instance reboot 
count to 1 (out of 4)
2015-06-19 20:54:50,573 DEBUG            sge:552  qstat: []
2015-06-19 20:54:51,227 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:54:51,227 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:54:51,303 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:54:51,303 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:54:51 
| last m_state change before 0:07:45 | time_rebooted before 0:00:32
2015-06-19 20:55:23,547 DEBUG            sge:552  qstat: []
2015-06-19 20:55:24,143 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:55:24,144 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:55:24,248 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:55:24,249 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:55:24 
| last m_state change before 0:08:17 | time_rebooted before 0:01:05
2015-06-19 20:55:56,497 DEBUG            sge:552  qstat: []
2015-06-19 20:55:57,061 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:55:57,061 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:55:57,124 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:55:57,124 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:55:57 
| last m_state change before 0:08:50 | time_rebooted before 0:01:38
2015-06-19 20:56:29,374 DEBUG            sge:552  qstat: []
2015-06-19 20:56:30,083 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:56:30,083 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:56:30,168 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:56:30,169 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:56:30 
| last m_state change before 0:09:23 | time_rebooted before 0:02:11
2015-06-19 20:57:02,427 DEBUG            sge:552  qstat: []
2015-06-19 20:57:03,010 DEBUG         master:2534 S&S: Migration..Completed; 
SGE..OK; FS object for transient_nfs..OK; FS object for galaxy..OK; FS object 
for galaxyIndices..OK; Postgres..OK; ProFTPd..OK; Galaxy..OK; 
GalaxyReports..OK; 
2015-06-19 20:57:03,010 DEBUG         master:2556 Have not heard from or 
checked on instance 'i-69a8a496' (IP: 54.164.165.200) for a while; checking now.
2015-06-19 20:57:03,073 DEBUG         master:2894 Requested instance 
'i-69a8a496' (IP: 54.164.165.200) update: old state: running; new state: running
2015-06-19 20:57:03,074 DEBUG         master:2659 'Maintaining' instance 
'i-69a8a496' (IP: 54.164.165.200) in 'running' state (last comm before 20:57:03 
| last m_state change before 0:09:56 | time_rebooted before 0:02:44
2015-06-19 20:57:41,299 DEBUG         master:1900 Checking for new version of 
CloudMan
2015-06-19 20:57:41,300 DEBUG           misc:668  Getting metadata 'revision' 
for file 'cm.tar.gz' from bucket 'cm-442caa28f69846327455b14f155f75cd'
2015-06-19 20:57:41,393 DEBUG           misc:668  Getting metadata 'revision' 
for file 'cm.tar.gz' from bucket 'cloudman'
2015-06-19 20:57:41,462 DEBUG         master:1907 Revision number for user's 
CloudMan: '9999999'; revision number for default CloudMan: 'None'
2015-06-19 20:57:41,659 DEBUG         master:1900 Checking for new version of 
CloudMan
2015-06-19 20:57:41,659 DEBUG           misc:668  Getting metadata 'revision' 
for file 'cm.tar.gz' from bucket 'cm-442caa28f69846327455b14f155f75cd'
2015-06-19 20:57:41,691 DEBUG           misc:668  Getting metadata 'revision' 
for file 'cm.tar.gz' from bucket 'cloudman'
2015-06-19 20:57:41,730 DEBUG         master:1907 Revision number for user's 
CloudMan: '9999999'; revision number for default CloudMan: 'None'
___________________________________________________________
Please keep all replies on the list by using "reply all"
in your mail client.  To manage your subscriptions to this
and other Galaxy lists, please use the interface at:
  https://lists.galaxyproject.org/

To search Galaxy mailing lists use the unified search at:
  http://galaxyproject.org/search/mailinglists/

Reply via email to