So with a patched leveldb to skip errors I now have a store.db that I can
extract the pg,mon,and osd map from. That said when I try to start kh10-8
it bombs out::
---------------------------------------
---------------------------------------
root@kh10-8:/var/lib/ceph/mon/ceph-kh10-8# ceph-mon -i $(hostname) -d
2016-08-13 22:30:54.596039 7fa8b9e088c0 0 ceph version 0.94.7
(d56bdf93ced6b80b07397d57e3fa68fe68304432), process ceph-mon, pid 708653
starting mon.kh10-8 rank 2 at 10.64.64.125:6789/0 mon_data
/var/lib/ceph/mon/ceph-kh10-8 fsid e452874b-cb29-4468-ac7f-f8901dfccebf
2016-08-13 22:30:54.608150 7fa8b9e088c0 0 starting mon.kh10-8 rank 2 at
10.64.64.125:6789/0 mon_data /var/lib/ceph/mon/ceph-kh10-8 fsid
e452874b-cb29-4468-ac7f-f8901dfccebf
2016-08-13 22:30:54.608395 7fa8b9e088c0 1 mon.kh10-8@-1(probing) e1
preinit fsid e452874b-cb29-4468-ac7f-f8901dfccebf
2016-08-13 22:30:54.608617 7fa8b9e088c0 1
mon.kh10-8@-1(probing).paxosservice(pgmap
0..35606392) refresh upgraded, format 0 -> 1
2016-08-13 22:30:54.608629 7fa8b9e088c0 1 mon.kh10-8@-1(probing).pg v0
on_upgrade discarding in-core PGMap
terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
what(): buffer::end_of_buffer
*** Caught signal (Aborted) **
in thread 7fa8b9e088c0
ceph version 0.94.7 (d56bdf93ced6b80b07397d57e3fa68fe68304432)
1: ceph-mon() [0x9b25ea]
2: (()+0x10330) [0x7fa8b8f0b330]
3: (gsignal()+0x37) [0x7fa8b73a8c37]
4: (abort()+0x148) [0x7fa8b73ac028]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fa8b7cb3535]
6: (()+0x5e6d6) [0x7fa8b7cb16d6]
7: (()+0x5e703) [0x7fa8b7cb1703]
8: (()+0x5e922) [0x7fa8b7cb1922]
9: ceph-mon() [0x853c39]
10:
(object_stat_collection_t::decode(ceph::buffer::list::iterator&)+0x167)
[0x894227]
11: (pg_stat_t::decode(ceph::buffer::list::iterator&)+0x5ff) [0x894baf]
12: (PGMap::update_pg(pg_t, ceph::buffer::list&)+0xa3) [0x91a8d3]
13: (PGMonitor::read_pgmap_full()+0x1d8) [0x68b9b8]
14: (PGMonitor::update_from_paxos(bool*)+0xbf7) [0x6977b7]
15: (PaxosService::refresh(bool*)+0x19a) [0x605b5a]
16: (Monitor::refresh_from_paxos(bool*)+0x1db) [0x5b1ffb]
17: (Monitor::init_paxos()+0x85) [0x5b2365]
18: (Monitor::preinit()+0x7d7) [0x5b6f87]
19: (main()+0x230c) [0x57853c]
20: (__libc_start_main()+0xf5) [0x7fa8b7393f45]
21: ceph-mon() [0x59a3c7]
2016-08-13 22:30:54.611791 7fa8b9e088c0 -1 *** Caught signal (Aborted) **
in thread 7fa8b9e088c0
ceph version 0.94.7 (d56bdf93ced6b80b07397d57e3fa68fe68304432)
1: ceph-mon() [0x9b25ea]
2: (()+0x10330) [0x7fa8b8f0b330]
3: (gsignal()+0x37) [0x7fa8b73a8c37]
4: (abort()+0x148) [0x7fa8b73ac028]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fa8b7cb3535]
6: (()+0x5e6d6) [0x7fa8b7cb16d6]
7: (()+0x5e703) [0x7fa8b7cb1703]
8: (()+0x5e922) [0x7fa8b7cb1922]
9: ceph-mon() [0x853c39]
10:
(object_stat_collection_t::decode(ceph::buffer::list::iterator&)+0x167)
[0x894227]
11: (pg_stat_t::decode(ceph::buffer::list::iterator&)+0x5ff) [0x894baf]
12: (PGMap::update_pg(pg_t, ceph::buffer::list&)+0xa3) [0x91a8d3]
13: (PGMonitor::read_pgmap_full()+0x1d8) [0x68b9b8]
14: (PGMonitor::update_from_paxos(bool*)+0xbf7) [0x6977b7]
15: (PaxosService::refresh(bool*)+0x19a) [0x605b5a]
16: (Monitor::refresh_from_paxos(bool*)+0x1db) [0x5b1ffb]
17: (Monitor::init_paxos()+0x85) [0x5b2365]
18: (Monitor::preinit()+0x7d7) [0x5b6f87]
19: (main()+0x230c) [0x57853c]
20: (__libc_start_main()+0xf5) [0x7fa8b7393f45]
21: ceph-mon() [0x59a3c7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
to interpret this.
--- begin dump of recent events ---
-33> 2016-08-13 22:30:54.593450 7fa8b9e088c0 5 asok(0x36a20f0)
register_command perfcounters_dump hook 0x365a050
-32> 2016-08-13 22:30:54.593480 7fa8b9e088c0 5 asok(0x36a20f0)
register_command 1 hook 0x365a050
-31> 2016-08-13 22:30:54.593486 7fa8b9e088c0 5 asok(0x36a20f0)
register_command perf dump hook 0x365a050
-30> 2016-08-13 22:30:54.593496 7fa8b9e088c0 5 asok(0x36a20f0)
register_command perfcounters_schema hook 0x365a050
-29> 2016-08-13 22:30:54.593499 7fa8b9e088c0 5 asok(0x36a20f0)
register_command 2 hook 0x365a050
-28> 2016-08-13 22:30:54.593501 7fa8b9e088c0 5 asok(0x36a20f0)
register_command perf schema hook 0x365a050
-27> 2016-08-13 22:30:54.593503 7fa8b9e088c0 5 asok(0x36a20f0)
register_command perf reset hook 0x365a050
-26> 2016-08-13 22:30:54.593505 7fa8b9e088c0 5 asok(0x36a20f0)
register_command config show hook 0x365a050
-25> 2016-08-13 22:30:54.593508 7fa8b9e088c0 5 asok(0x36a20f0)
register_command config set hook 0x365a050
-24> 2016-08-13 22:30:54.593510 7fa8b9e088c0 5 asok(0x36a20f0)
register_command config get hook 0x365a050
-23> 2016-08-13 22:30:54.593512 7fa8b9e088c0 5 asok(0x36a20f0)
register_command config diff hook 0x365a050
-22> 2016-08-13 22:30:54.593513 7fa8b9e088c0 5 asok(0x36a20f0)
register_command log flush hook 0x365a050
-21> 2016-08-13 22:30:54.593557 7fa8b9e088c0 5 asok(0x36a20f0)
register_command log dump hook 0x365a050
-20> 2016-08-13 22:30:54.593561 7fa8b9e088c0 5 asok(0x36a20f0)
register_command log reopen hook 0x365a050
-19> 2016-08-13 22:30:54.596039 7fa8b9e088c0 0 ceph version 0.94.7
(d56bdf93ced6b80b07397d57e3fa68fe68304432), process ceph-mon, pid 708653
-18> 2016-08-13 22:30:54.597587 7fa8b9e088c0 5 asok(0x36a20f0) init
/var/run/ceph/ceph-mon.kh10-8.asok
-17> 2016-08-13 22:30:54.597601 7fa8b9e088c0 5 asok(0x36a20f0)
bind_and_listen /var/run/ceph/ceph-mon.kh10-8.asok
-16> 2016-08-13 22:30:54.597767 7fa8b9e088c0 5 asok(0x36a20f0)
register_command 0 hook 0x36560c0
-15> 2016-08-13 22:30:54.597775 7fa8b9e088c0 5 asok(0x36a20f0)
register_command version hook 0x36560c0
-14> 2016-08-13 22:30:54.597778 7fa8b9e088c0 5 asok(0x36a20f0)
register_command git_version hook 0x36560c0
-13> 2016-08-13 22:30:54.597781 7fa8b9e088c0 5 asok(0x36a20f0)
register_command help hook 0x365a150
-12> 2016-08-13 22:30:54.597783 7fa8b9e088c0 5 asok(0x36a20f0)
register_command get_command_descriptions hook 0x365a140
-11> 2016-08-13 22:30:54.597860 7fa8b5181700 5 asok(0x36a20f0) entry
start
-10> 2016-08-13 22:30:54.608150 7fa8b9e088c0 0 starting mon.kh10-8 rank
2 at 10.64.64.125:6789/0 mon_data /var/lib/ceph/mon/ceph-kh10-8 fsid
e452874b-cb29-4468-ac7f-f8901dfccebf
-9> 2016-08-13 22:30:54.608210 7fa8b9e088c0 1 -- 10.64.64.125:6789/0
learned my addr 10.64.64.125:6789/0
-8> 2016-08-13 22:30:54.608214 7fa8b9e088c0 1 accepter.accepter.bind
my_inst.addr is 10.64.64.125:6789/0 need_addr=0
-7> 2016-08-13 22:30:54.608279 7fa8b9e088c0 5 adding auth protocol:
cephx
-6> 2016-08-13 22:30:54.608282 7fa8b9e088c0 5 adding auth protocol:
cephx
-5> 2016-08-13 22:30:54.608311 7fa8b9e088c0 10 log_channel(cluster)
update_config to_monitors: true to_syslog: false syslog_facility: daemon
prio: info)
-4> 2016-08-13 22:30:54.608317 7fa8b9e088c0 10 log_channel(audit)
update_config to_monitors: true to_syslog: false syslog_facility: local0
prio: info)
-3> 2016-08-13 22:30:54.608395 7fa8b9e088c0 1 mon.kh10-8@-1(probing)
e1 preinit fsid e452874b-cb29-4468-ac7f-f8901dfccebf
-2> 2016-08-13 22:30:54.608617 7fa8b9e088c0 1
mon.kh10-8@-1(probing).paxosservice(pgmap
0..35606392) refresh upgraded, format 0 -> 1
-1> 2016-08-13 22:30:54.608629 7fa8b9e088c0 1 mon.kh10-8@-1(probing).pg
v0 on_upgrade discarding in-core PGMap
0> 2016-08-13 22:30:54.611791 7fa8b9e088c0 -1 *** Caught signal
(Aborted) **
in thread 7fa8b9e088c0
ceph version 0.94.7 (d56bdf93ced6b80b07397d57e3fa68fe68304432)
1: ceph-mon() [0x9b25ea]
2: (()+0x10330) [0x7fa8b8f0b330]
3: (gsignal()+0x37) [0x7fa8b73a8c37]
4: (abort()+0x148) [0x7fa8b73ac028]
5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fa8b7cb3535]
6: (()+0x5e6d6) [0x7fa8b7cb16d6]
7: (()+0x5e703) [0x7fa8b7cb1703]
8: (()+0x5e922) [0x7fa8b7cb1922]
9: ceph-mon() [0x853c39]
10:
(object_stat_collection_t::decode(ceph::buffer::list::iterator&)+0x167)
[0x894227]
11: (pg_stat_t::decode(ceph::buffer::list::iterator&)+0x5ff) [0x894baf]
12: (PGMap::update_pg(pg_t, ceph::buffer::list&)+0xa3) [0x91a8d3]
13: (PGMonitor::read_pgmap_full()+0x1d8) [0x68b9b8]
14: (PGMonitor::update_from_paxos(bool*)+0xbf7) [0x6977b7]
15: (PaxosService::refresh(bool*)+0x19a) [0x605b5a]
16: (Monitor::refresh_from_paxos(bool*)+0x1db) [0x5b1ffb]
17: (Monitor::init_paxos()+0x85) [0x5b2365]
18: (Monitor::preinit()+0x7d7) [0x5b6f87]
19: (main()+0x230c) [0x57853c]
20: (__libc_start_main()+0xf5) [0x7fa8b7393f45]
21: ceph-mon() [0x59a3c7]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed
to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
-2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 10000
max_new 1000
log_file
--- end dump of recent events ---
Aborted (core dumped)
---------------------------------------
---------------------------------------
I feel like I am so close but so far. Can anyone give me a nudge as to what
I can do next? it looks like it is bombing out on trying to get an updated
paxos.
On Fri, Aug 12, 2016 at 1:09 PM, Sean Sullivan <seapasu...@uchicago.edu>
wrote:
> A coworker patched leveldb and we were able to export quite a bit of data
> from kh08's leveldb database. At this point I think I need to re-construct
> a new leveldb with whatever values I can. Is it the same leveldb database
> across all 3 montiors? IE will keys exported from one work in the other?
> All should have the same keys/values although constructed differently
> right? I can't blindly copy /var/lib/ceph/mon/ceph-$(hostname)/store.db/
> from one host to another right? But can I copy the keys/values from one to
> another?
>
> On Fri, Aug 12, 2016 at 12:45 PM, Sean Sullivan <seapasu...@uchicago.edu>
> wrote:
>
>> ceph-monstore-tool? Is that the same as monmaptool? oops! NM found it in
>> ceph-test package::
>>
>> I can't seem to get it working :-( dump monmap or any of the commands.
>> They all bomb out with the same message:
>>
>> root@kh10-8:/var/lib/ceph/mon/ceph-kh10-8/store.db# ceph-monstore-tool
>> /var/lib/ceph/mon/ceph-kh10-8 dump-trace -- /tmp/test.trace
>> Corruption: 1 missing files; e.g.: /var/lib/ceph/mon/ceph-kh10-8/
>> store.db/10882319.ldb
>> root@kh10-8:/var/lib/ceph/mon/ceph-kh10-8/store.db# ceph-monstore-tool
>> /var/lib/ceph/mon/ceph-kh10-8 dump-keys
>> Corruption: 1 missing files; e.g.: /var/lib/ceph/mon/ceph-kh10-8/
>> store.db/10882319.ldb
>>
>>
>> I need to clarify as I originally had 2 clusters with this issue and now
>> I have 1 with all 3 monitors dead and 1 that I was successfully able to
>> repair. I am about to recap everything I know about the issue and the issue
>> at hand. Should I start a new email thread about this instead?
>>
>> The cluster that is currently having issues is on hammer (94.7), and the
>> monitor stats are the same::
>> root@kh08-8:~# cat /proc/cpuinfo | grep -iE "model name" | uniq -c
>> 24 model name : Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz
>> ext4 volume comprised of 4x300GB 10k drives in raid 10.
>> ubuntu 14.04
>>
>> root@kh08-8:~# uname -a
>> Linux kh08-8 3.13.0-76-generic #120-Ubuntu SMP Mon Jan 18 15:59:10 UTC
>> 2016 x86_64 x86_64 x86_64 GNU/Linux
>> root@kh08-8:~# ceph --version
>> ceph version 0.94.7 (d56bdf93ced6b80b07397d57e3fa68fe68304432)
>>
>>
>> From here: Here are the errors I am getting when starting each of the
>> monitors::
>>
>>
>> ---------------
>> root@kh08-8:~# /usr/bin/ceph-mon --cluster=ceph -i kh08-8 -d
>> 2016-08-11 22:15:23.731550 7fe5ad3e98c0 0 ceph version 0.94.7
>> (d56bdf93ced6b80b07397d57e3fa68fe68304432), process ceph-mon, pid 317309
>> Corruption: error in middle of record
>> 2016-08-11 22:15:28.274340 7fe5ad3e98c0 -1 error opening mon data
>> directory at '/var/lib/ceph/mon/ceph-kh08-8': (22) Invalid argument
>> --
>> root@kh09-8:~# /usr/bin/ceph-mon --cluster=ceph -i kh09-8 -d
>> 2016-08-11 22:14:28.252370 7f7eaab908c0 0 ceph version 0.94.7
>> (d56bdf93ced6b80b07397d57e3fa68fe68304432), process ceph-mon, pid 308888
>> Corruption: 14 missing files; e.g.: /var/lib/ceph/mon/ceph-kh09-8/
>> store.db/10845998.ldb
>> 2016-08-11 22:14:35.094237 7f7eaab908c0 -1 error opening mon data
>> directory at '/var/lib/ceph/mon/ceph-kh09-8': (22) Invalid argument
>> --
>> root@kh10-8:/var/lib/ceph/mon/ceph-kh10-8/store.db# /usr/bin/ceph-mon
>> --cluster=ceph -i kh10-8 -d
>> 2016-08-11 22:17:54.632762 7f80bf34d8c0 0 ceph version 0.94.7
>> (d56bdf93ced6b80b07397d57e3fa68fe68304432), process ceph-mon, pid 292620
>> Corruption: 1 missing files; e.g.: /var/lib/ceph/mon/ceph-kh10-8/
>> store.db/10882319.ldb
>> 2016-08-11 22:18:01.207749 7f80bf34d8c0 -1 error opening mon data
>> directory at '/var/lib/ceph/mon/ceph-kh10-8': (22) Invalid argument
>> ---------------
>>
>>
>> for kh08, a coworker patched leveldb to print and skip on the first error
>> and that one is also missing a bunch of files. As such I think kh10-8 is my
>> most likely candidate to recover but either way recovery is probably not an
>> option. I see leveldb has a repair.cc (https://github.com/google/lev
>> eldb/blob/master/db/repair.cc)) but I do not see repair mentioned in
>> monitor in respect to the dbstore. I tried using the leveldb python module
>> (plyvel) to attempt a repair but my repl just ends up dying.
>>
>> I understand two things:: 1.) Without rebuilding the monitor backend
>> leveldb (the cluster map as I understand it) store all of the data in the
>> cluster is essentialy lost (right?)
>> 2.) it is possible to rebuild
>> this database via some form of magic or (source)ry as all of this data is
>> essential held throughout the cluster as well.
>>
>> We only use radosgw / S3 for this cluster. If there is a way to recover
>> my data that is easier//more likely than rebuilding the leveldb of a
>> monitor and starting a single monitor cluster up I would like to switch
>> gears and focus on that.
>>
>> Looking at the dev docs:
>> http://docs.ceph.com/docs/hammer/architecture/#cluster-map
>> it has 5 main parts::
>>
>> ```
>> The Monitor Map: Contains the cluster fsid, the position, name address
>> and port of each monitor. It also indicates the current epoch, when the map
>> was created, and the last time it changed. To view a monitor map, execute
>> ceph mon dump.
>> The OSD Map: Contains the cluster fsid, when the map was created and last
>> modified, a list of pools, replica sizes, PG numbers, a list of OSDs and
>> their status (e.g., up, in). To view an OSD map, execute ceph osd dump.
>> The PG Map: Contains the PG version, its time stamp, the last OSD map
>> epoch, the full ratios, and details on each placement group such as the PG
>> ID, the Up Set, the Acting Set, the state of the PG (e.g., active + clean),
>> and data usage statistics for each pool.
>> The CRUSH Map: Contains a list of storage devices, the failure domain
>> hierarchy (e.g., device, host, rack, row, room, etc.), and rules for
>> traversing the hierarchy when storing data. To view a CRUSH map, execute
>> ceph osd getcrushmap -o {filename}; then, decompile it by executing
>> crushtool -d {comp-crushmap-filename} -o {decomp-crushmap-filename}. You
>> can view the decompiled map in a text editor or with cat.
>> The MDS Map: Contains the current MDS map epoch, when the map was
>> created, and the last time it changed. It also contains the pool for
>> storing metadata, a list of metadata servers, and which metadata servers
>> are up and in. To view an MDS map, execute ceph mds dump.
>> ```
>>
>> As we don't use cephfs mds can essentially be blank(right) so I am left
>> with 4 valid maps needed to get a working cluster again. I don't see auth
>> mentioned in there but that too. Then I just need to rebuild the leveldb
>> database somehow with the right information and I should be good. So long
>> long long journey ahead.
>>
>> I don't think that the data is stored in strings or json, right? Am I
>> going down the wrong path here? Is there a shorter/simpler path to retrieve
>> the data from a cluster that lost all 3 monitors in power falure? If I am
>> going down the right path is there any advice on how I can assemble/repair
>> the database?
>>
>> I see that there is a rbd recovery from a dead cluster tool. Is it
>> possible to do the same with s3 objects?
>>
>> On Thu, Aug 11, 2016 at 11:15 AM, Wido den Hollander <w...@42on.com>
>> wrote:
>>
>>>
>>> > Op 11 augustus 2016 om 15:17 schreef Sean Sullivan <
>>> seapasu...@uchicago.edu>:
>>> >
>>> >
>>> > Hello Wido,
>>> >
>>> > Thanks for the advice. While the data center has a/b circuits and
>>> > redundant power, etc if a ground fault happens it travels outside and
>>> > fails causing the whole building to fail (apparently).
>>> >
>>> > The monitors are each the same with
>>> > 2x e5 cpus
>>> > 64gb of ram
>>> > 4x 300gb 10k SAS drives in raid 10 (write through mode).
>>> > Ubuntu 14.04 with the latest updates prior to power failure
>>> (2016/Aug/10 -
>>> > 3am CST)
>>> > Ceph hammer LTS 0.94.7
>>> >
>>> > (we are still working on our jewel test cluster so it is planned but
>>> not in
>>> > place yet)
>>> >
>>> > The only thing that seems to be corrupt is the monitors leveldb
>>> store. I
>>> > see multiple issues on Google leveldb github from March 2016 about
>>> fsync
>>> > and power failure so I assume this is an issue with leveldb.
>>> >
>>> > I have backed up /var/lib/ceph/Mon on all of my monitors before trying
>>> to
>>> > proceed with any form of recovery.
>>> >
>>> > Is there any way to reconstruct the leveldb or replace the monitors and
>>> > recover the data?
>>> >
>>> I don't know. I have never done it. Other people might know this better
>>> than me.
>>>
>>> Maybe 'ceph-monstore-tool' can help you?
>>>
>>> Wido
>>>
>>> > I found the following post in which sage says it is tedious but
>>> possible. (
>>> > http://www.spinics.net/lists/ceph-devel/msg06662.html). Tedious is
>>> fine if
>>> > I have any chance of doing it. I have the fsid, the Mon key map and
>>> all of
>>> > the osds look to be fine so all of the previous osd maps are there.
>>> >
>>> > I just don't understand what key/values I need inside.
>>> >
>>> > On Aug 11, 2016 1:33 AM, "Wido den Hollander" <w...@42on.com> wrote:
>>> >
>>> > >
>>> > > > Op 11 augustus 2016 om 0:10 schreef Sean Sullivan <
>>> > > seapasu...@uchicago.edu>:
>>> > > >
>>> > > >
>>> > > > I think it just got worse::
>>> > > >
>>> > > > all three monitors on my other cluster say that ceph-mon can't open
>>> > > > /var/lib/ceph/mon/$(hostname). Is there any way to recover if you
>>> lose
>>> > > all
>>> > > > 3 monitors? I saw a post by Sage saying that the data can be
>>> recovered as
>>> > > > all of the data is held on other servers. Is this possible? If so
>>> has
>>> > > > anyone had any experience doing so?
>>> > >
>>> > > I have never done so, so I couldn't tell you.
>>> > >
>>> > > However, it is weird that on all three it got corrupted. What
>>> hardware are
>>> > > you using? Was it properly protected against power failure?
>>> > >
>>> > > If you mon store is corrupted I'm not sure what might happen.
>>> > >
>>> > > However, make a backup of ALL monitors right now before doing
>>> anything.
>>> > >
>>> > > Wido
>>> > >
>>> > > > _______________________________________________
>>> > > > ceph-users mailing list
>>> > > > ceph-users@lists.ceph.com
>>> > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>> > >
>>>
>>
>>
>>
>> --
>> - Sean: I wrote this. -
>>
>
>
>
> --
> - Sean: I wrote this. -
>
--
- Sean: I wrote this. -
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
