Re: [ceph-users] Troubleshooting incomplete PG's

2017-04-01 Thread Brad Hubbard 
On Fri, Mar 31, 2017 at 5:19 AM, nokia ceph  wrote:
> Hello Brad,
>
> Many thanks of the info :)
>
> ENV:-- Kracken - bluestore - EC 4+1 - 5 node cluster : RHEL7
>
> What is the status of the down+out osd? Only one osd osd.6 down and out from
> cluster.
> What role did/does it play? Mostimportantly, is it osd.6? Yes, due to
> underlying I/O error issue we removed this device from the cluster.
>
> I put this parameter " osd_find_best_info_ignore_history_les = true" in
> ceph.conf, and find those 22 PG's were changed to "down+remapped" . Now all
> are reverted to "remapped+incomplete" state.
>
> #ceph pg stat 2> /dev/null
> v2731828: 4096 pgs: 1 incomplete, 21 remapped+incomplete, 4074 active+clean;
> 268 TB data, 371 TB used, 267 TB / 638 TB avail
>
> ## ceph -s
> 2017-03-30 19:02:14.350242 7f8b0415f700 -1 WARNING: the following dangerous
> and experimental features are enabled: bluestore,rocksdb
> 2017-03-30 19:02:14.366545 7f8b0415f700 -1 WARNING: the following dangerous
> and experimental features are enabled: bluestore,rocksdb
> cluster bd8adcd0-c36d-4367-9efe-f48f5ab5f108
>  health HEALTH_ERR
> 22 pgs are stuck inactive for more than 300 seconds
> 22 pgs incomplete
> 22 pgs stuck inactive
> 22 pgs stuck unclean
>  monmap e2: 5 mons at
> {au-adelaide=10.50.21.24:6789/0,au-brisbane=10.50.21.22:6789/0,au-canberra=10.50.21.23:6789/0,au-melbourne=10.50.21.21:6789/0,au-sydney=10.50.21.20:6789/0}
> election epoch 180, quorum 0,1,2,3,4

Are you *actually* trying to create a cluster that is as
geographically dispersed as these machine names indicate?

> au-sydney,au-melbourne,au-brisbane,au-canberra,au-adelaide
> mgr active: au-adelaide
>  osdmap e6506: 117 osds: 117 up, 117 in; 21 remapped pgs
> flags sortbitwise,require_jewel_osds,require_kraken_osds
>   pgmap v2731828: 4096 pgs, 1 pools, 268 TB data, 197 Mobjects
> 371 TB used, 267 TB / 638 TB avail
> 4074 active+clean
>   21 remapped+incomplete
>1 incomplete
>
>
> ## ceph osd dump 2>/dev/null | grep cdvr
> pool 1 'cdvr_ec' erasure size 5 min_size 4 crush_ruleset 1 object_hash
> rjenkins pg_num 4096 pgp_num 4096 last_change 456 flags
> hashpspool,nodeep-scrub stripe_width 65536
>
> Inspecting affected PG 1.e4b
>
> # ceph pg dump 2> /dev/null | grep 1.e4b
> 1.e4b 50832  00 0   0 73013340821
> 1000610006 remapped+incomplete 2017-03-30 14:14:26.297098 3844'161662
> 6506:325748 [113,66,15,73,103]113  [NONE,NONE,NONE,73,NONE]
> 73 1643'139486 2017-03-21 04:56:16.683953 0'0 2017-02-21
> 10:33:50.012922
>
> When I trigger below command.
>
> #ceph pg force_create_pg 1.e4b
> pg 1.e4b now creating, ok
>
> As it went to creating state, no change after that. Can you explain why this
> PG showing null values after triggering "force_create_pg",?
>
> ]# ceph pg dump 2> /dev/null | grep 1.e4b
> 1.e4b 0  00 0   0   0
> 00creating 2017-03-30 19:07:00.982178 0'0
> 0:0 [] -1[] -1
> 0'0   0.00 0'0   0.00
>
> Then I triggered below command
>
> # ceph pg  repair 1.e4b
> Error EAGAIN: pg 1.e4b has no primary osd  --<<
>
> Could you please provide answer for below queries.
>
> 1. How to fix this "incomplete+remapped" PG issue, here all OSD's were up
> and running and affected OSD marked out and removed from the cluster.
> 2. Will reduce min_size helps? currently it set to 4. Could you please
> explain what is the impact if we reduce min_size for the current config EC
> 4+1
> 3. Is there any procedure to safely remove an affected PG? As per my
> understanding I'm aware about this command.
>
> ===
> #ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph --pgid 1.e4b --op
> remove
> ===
>
> Awaiting for your suggestions to proceed.
>
> Thanks
>
>
>
>
>
>
> On Thu, Mar 30, 2017 at 7:32 AM, Brad Hubbard  wrote:
>>
>>
>>
>> On Thu, Mar 30, 2017 at 4:53 AM, nokia ceph 
>> wrote:
>> > Hello,
>> >
>> > Env:-
>> > 5 node, EC 4+1 bluestore kraken v11.2.0 , RHEL7.2
>> >
>> > As part of our resillency testing with kraken bluestore, we face more
>> > PG's
>> > were in incomplete+remapped state. We tried to repair each PG using
>> > "ceph pg
>> > repair " still no luck. Then we planned to remove incomplete PG's
>> > using below procedure.
>> >
>> >
>> > #ceph health detail | grep  1.e4b
>> > pg 1.e4b is remapped+incomplete, acting [2147483647,66,15,73,2147483647]
>> > (reducing pool cdvr_ec min_size from 4 may help; search ceph.com/docs
>> > for
>> > 'incomplete')
>>
>> "Incomplete Ceph detects that a placement group is missing information
>> about
>> writes that may have occurred, or does not have any healthy

Re: [ceph-users] FreeBSD port net/ceph-devel released

2017-04-01 Thread Willem Jan Withagen 
On 1-4-2017 21:59, Wido den Hollander wrote:
> 
>> Op 31 maart 2017 om 19:15 schreef Willem Jan Withagen :
>>
>>
>> On 31-3-2017 17:32, Wido den Hollander wrote:
>>> Hi Willem Jan,
>>>
 Op 30 maart 2017 om 13:56 schreef Willem Jan Withagen
 :


 Hi,

 I'm pleased to announce that my efforts to port to FreeBSD have
 resulted in a ceph-devel port commit in the ports tree.

 https://www.freshports.org/net/ceph-devel/

>>>
>>> Awesome work! I don't touch FreeBSD that much, but I can imagine that
>>> people want this.
>>>
>>> Out of curiosity, does this run on ZFS under FreeBSD? Or what
>>> Filesystem would you use behind FileStore with this? Or does
>>> BlueStore work?
>>
>> Since I'm a huge ZFS fan, that is what I run it on.
> 
> Cool! The ZIL, ARC and L2ARC can actually make that very fast. Interesting!

Right, ZIL is magic, and more or equal to the journal now used with OSDs
for exactly the same reason. Sad thing is that a write is now 3*
journaled: 1* by Ceph, and 2* by ZFS. Which means that the used
bandwidth to the SSDs is double of what it could be.

Had some discussion about this, but disabling the Ceph journal is not
just setting an option. Although I would like to test performance of an
OSD with just the ZFS journal. But I expect that the OSD journal is
rather firmly integrated.

Now the real nice thing is that one does not need to worry about
cacheing the OSD performance. This is fully covered by ZFS. Both by ARC
and L2ARC. And ZIL and L2ARC can be constructed again in all shapes and
forms that all AFS vdev's can be made.
So for the ZIL you'd build and SSD's mirror: double the write speed, but
still redundant. For L2ARC I'd concatenate 2 SSD's to get the read
bandwidth. And contrary to some of the other caches ZFS does not return
errors if the l2arc devices go down. (note that data errors are detected
by checksumming) So that again is one less thing to worry about.

> CRC and Compression from ZFS are also very nice.

I did not want to go into too much details, but this is a large part of
the reasons. Compression I tried a bit, but does cost quite a bit of
performance at the Ceph end. Perhaps because the write to the journal is
synced, and thus has to way on both compression and synced writting.

It also bring snapshots without much hassle. But I have not yet figured
(looked at) out if and how btrfs snapshots are used.

Other challenge is the Ceph deep scrubbing: checking for corruption
within files. ZFS is able to detect corruption all by itself due to
extensive file checksumming. And with something way much stronger/better
that crc32. (just put on my fireproof suite)
So I'm not certain that deep-scrub would be obsolete, but I think it
could the frequency could perhaps go down, and/or be triggered by ZFS
errors after scrubbing a pool. Something that has way much less impact
on performance.

In some of the talks I give, I always try to explain to people that RAID
and RAID controllers are the current dinosaurs of IT.

>> To be honest I have not tested on UFS, but I would expect that the xattr
>> are not long enough.
>>
>> BlueStore is not (yet) available because there is a different AIO
>> implementation on FreeBSD. But Sage thinks it is very doable to glue in
>> posix AIO. And one of my port reviewers has offered to look at it. So it
>> could be that BlueStore will be available in the foreseeable future.
>>
>> --WjW

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] FreeBSD port net/ceph-devel released

2017-04-01 Thread Wido den Hollander 

> Op 31 maart 2017 om 19:15 schreef Willem Jan Withagen :
> 
> 
> On 31-3-2017 17:32, Wido den Hollander wrote:
> > Hi Willem Jan,
> > 
> >> Op 30 maart 2017 om 13:56 schreef Willem Jan Withagen
> >> :
> >> 
> >> 
> >> Hi,
> >> 
> >> I'm pleased to announce that my efforts to port to FreeBSD have
> >> resulted in a ceph-devel port commit in the ports tree.
> >> 
> >> https://www.freshports.org/net/ceph-devel/
> >> 
> > 
> > Awesome work! I don't touch FreeBSD that much, but I can imagine that
> > people want this.
> > 
> > Out of curiosity, does this run on ZFS under FreeBSD? Or what
> > Filesystem would you use behind FileStore with this? Or does
> > BlueStore work?
> 
> Since I'm a huge ZFS fan, that is what I run it on.

Cool! The ZIL, ARC and L2ARC can actually make that very fast. Interesting!

CRC and Compression from ZFS are also very nice.

> To be honest I have not tested on UFS, but I would expect that the xattr
> are not long enough.
> 
> BlueStore is not (yet) available because there is a different AIO
> implementation on FreeBSD. But Sage thinks it is very doable to glue in
> posix AIO. And one of my port reviewers has offered to look at it. So it
> could be that BlueStore will be available in the foreseeable future.
>
> --WjW
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] clock skew

2017-04-01 Thread Wido den Hollander 

> Op 1 april 2017 om 16:02 schreef John Petrini :
> 
> 
> Hello,
> 
> I'm also curious about the impact of clock drift. We see the same on both
> of our clusters despite trying various NTP servers including our own local
> servers. Ultimately we just ended up adjusting our monitoring to be less
> sensitive to it since the clock drift always resolves on its own. Is this a
> dangerous practice?

It can cause the Monitors to not being able to elect. Very far clock drifts can 
even cause cephx problems.

So yes, it can lead to downtime of your cluster.

The 50ms clock drift it allows by default should be enough and NTP should keep 
it below that.

Wido

> 
> ___
> 
> John Petrini
> 
> NOC Systems Administrator   //   *CoreDial, LLC*   //   coredial.com
> //   [image:
> Twitter]    [image: LinkedIn]
>    [image: Google Plus]
>    [image: Blog]
> 
> Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
> *P: *215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
> jpetr...@coredial.com
> 
> 
> Interested in sponsoring PartnerConnex 2017? Learn more.
> 
> 
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission,  dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
> 
> On Sat, Apr 1, 2017 at 9:12 AM, mj  wrote:
> 
> > Hi,
> >
> > On 04/01/2017 02:10 PM, Wido den Hollander wrote:
> >
> >> You could try the chrony NTP daemon instead of ntpd and make sure all
> >> MONs are peers from each other.
> >>
> > I understand now what that means. I have set it up according to your
> > suggestion.
> >
> > Curious to see how this works out, thanks!
> >
> >
> > MJ
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] v11.2.0 OSD crashing "src/os/bluestore/KernelDevice.cc: 541: FAILED assert((uint64_t)r == len) "

2017-04-01 Thread nokia ceph 
Hello,

We are getting below trace on failed OSD's . Can you please explain from
the below code why this issue happening. We suspect it could be because of
underlying HW issue. We can't find anything from the syslogs. All the OSD
disk are in healthy condition.

Link :- https://fossies.org/linux/ceph/src/os/bluestore/KernelDevice.cc

===
2017-03-31 03:51:28.262403 7f71f3ed5940 -1
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/11.2.0/rpm/el7/BUILD/ceph-11.2.0/src/os/bluestore/
KernelDevice.cc: In function 'virtual int KernelDevice::read(uint64_t,
uint64_t, ceph::bufferlist*, IOContext*, bool)' thread 7f71f3ed5940 time
2017-03-31 03:51:28.256298
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/11.2.0/rpm/el7/BUILD/ceph-11.2.0/*src/os/bluestore/KernelDevice.cc:
541: FAILED assert((uint64_t)r == len)  =>*

ceph version 11.2.0 (f223e27eeb35991352ebc1f67423d4ebc252adb7)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x85) [0x7f71f49b6b35]
2: (KernelDevice::read(unsigned long, unsigned long, ceph::buffer::list*,
IOContext*, bool)+0x7a0) [0x7f71f47fdf10]
3: (BlueFS::_read(BlueFS::FileReader*, BlueFS::FileReaderBuffer*, unsigned
long, unsigned long, ceph::buffer::list*, char*)+0x47a) [0x7f71f47c8d4a]
4: (BlueRocksSequentialFile::Read(unsigned long, rocksdb::Slice*,
char*)+0x36) [0x7f71f47ecd46]
5: (rocksdb::SequentialFileReader::Read(unsigned long, rocksdb::Slice*,
char*)+0x16) [0x7f71f48f4136]
6: (rocksdb::log::Reader::ReadMore(unsigned long*, int*)+0xd8)
[0x7f71f487d658]
7: (rocksdb::log::Reader::ReadPhysicalRecord(rocksdb::Slice*, unsigned
long*)+0x44) [0x7f71f487d734]
8: (rocksdb::log::Reader::ReadRecord(rocksdb::Slice*, std::string*,
rocksdb::WALRecoveryMode)+0xe8) [0x7f71f487d9e8]
9: (rocksdb::DBImpl::RecoverLogFiles(std::vector > const&, unsigned long*, bool)+0xdc8)
[0x7f71f4837a18]
10: (rocksdb::DBImpl::Recover(std::vector const&, bool, bool,
bool)+0x866) [0x7f71f4839386]
11: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std
===

Thanks
Jayaram
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] clock skew

2017-04-01 Thread John Petrini 
Just ntp.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] clock skew

2017-04-01 Thread mj 



On 04/01/2017 04:02 PM, John Petrini wrote:

Hello,

I'm also curious about the impact of clock drift. We see the same on
both of our clusters despite trying various NTP servers including our
own local servers. Ultimately we just ended up adjusting our monitoring
to be less sensitive to it since the clock drift always resolves on its
own. Is this a dangerous practice?


Are you running ntp, or this chrony? (which I did not know of)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] clock skew

2017-04-01 Thread John Petrini 
Hello,

I'm also curious about the impact of clock drift. We see the same on both
of our clusters despite trying various NTP servers including our own local
servers. Ultimately we just ended up adjusting our monitoring to be less
sensitive to it since the clock drift always resolves on its own. Is this a
dangerous practice?

___

John Petrini

NOC Systems Administrator   //   *CoreDial, LLC*   //   coredial.com
//   [image:
Twitter]    [image: LinkedIn]
   [image: Google Plus]
   [image: Blog]

Hillcrest I, 751 Arbor Way, Suite 150, Blue Bell PA, 19422
*P: *215.297.4400 x232   //   *F: *215.297.4401   //   *E: *
jpetr...@coredial.com


Interested in sponsoring PartnerConnex 2017? Learn more.


The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission,  dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.

On Sat, Apr 1, 2017 at 9:12 AM, mj  wrote:

> Hi,
>
> On 04/01/2017 02:10 PM, Wido den Hollander wrote:
>
>> You could try the chrony NTP daemon instead of ntpd and make sure all
>> MONs are peers from each other.
>>
> I understand now what that means. I have set it up according to your
> suggestion.
>
> Curious to see how this works out, thanks!
>
>
> MJ
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] clock skew

2017-04-01 Thread mj 

Hi,

On 04/01/2017 02:10 PM, Wido den Hollander wrote:

You could try the chrony NTP daemon instead of ntpd and make sure all
MONs are peers from each other.
I understand now what that means. I have set it up according to your 
suggestion.


Curious to see how this works out, thanks!

MJ
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Apply for an official mirror at CN

2017-04-01 Thread SJ Zhu 
On Sat, Apr 1, 2017 at 8:10 PM, Wido den Hollander  wrote:
> Great! Very good to hear. We can CNAME cn.ceph.com to that location?


Yes, please CNAME to mirrors.ustc.edu.cn, and I will set vhost in our
nginx for the
ceph directory.

Thanks

-- 
Regards,
Shengjing Zhu
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] clock skew

2017-04-01 Thread mj 

Hi Wido,

On 04/01/2017 02:10 PM, Wido den Hollander wrote:

That warning is there for a reason. I suggest you double-check your NTP and 
clocks on the machines. This should never happen in production.
I know... Don't understand why this happens..! Tried both ntpd and 
systemd-timesyncd. I did not yet know chrony, will try it.


I imagined that a 0.2 sec time skew would not be too disasterous.. As a 
side note: I cannot find explained anywhere WHAT could happen if the 
skew becomes too big. Only that we should prevent it. (data loss?)



Are you running the MONs inside Virtual Machines? They are more likely to have 
drifting clocks.

Nope. All bare metal on new supermicro servers.


You could try the chrony NTP daemon instead of ntpd and make sure all MONs are 
peers from each other.

Will try that.

I had set all MONs to sync with chime1.surfnet.nl - chime4. We usually 
have very good experiences with those ntp servers.


So, you're telling me that the MONs should be peers from each other... 
But if all MONs listen/sync to/with each other, where do I configure the 
external stratum1 source.?


MJ
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Apply for an official mirror at CN

2017-04-01 Thread Wido den Hollander 

> Op 1 april 2017 om 12:20 schreef SJ Zhu :
> 
> 
> Hi all,
> 
> We'd like to apply for the official mirror at China.
> We have run the ceph mirror for more than one year, at
> https://mirrors.ustc.edu.cn/ceph/
> 
> The server infomation is:
> * located at USTC(University of Science and Technology of China)
> * 2Gbit connection
> * both IPv4/IPv6
> * 60TB disk
> * rsync service at rsync://rsync.mirrors.ustc.edu.cn/ceph/
> * contact: lug(at)ustc.edu.cn
> 

Great! Very good to hear. We can CNAME cn.ceph.com to that location?

Wido

> Thanks,
> Shengjing Zhu
> 
> --
> Linux User Group
> University of Science and Technology of China
> Homepage: https://lug.ustc.edu.cn/
> E-Mail: l...@ustc.edu.cn
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] clock skew

2017-04-01 Thread Wido den Hollander 

> Op 1 april 2017 om 11:17 schreef mj :
> 
> 
> Hi,
> 
> Despite ntp, we keep getting clock skews that auto disappear again after 
> a few minutes.
> 
> To prevent the unneccerasy HEALTH_WARNs, I have increased the 
> mon_clock_drift_allowed to 0.2, as can be seen below:
> 

That warning is there for a reason. I suggest you double-check your NTP and 
clocks on the machines. This should never happen in production.

Are you running the MONs inside Virtual Machines? They are more likely to have 
drifting clocks.

You could try the chrony NTP daemon instead of ntpd and make sure all MONs are 
peers from each other.

Wido

> > root@ceph1:~# ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show 
> > | grep clock
> > "mon_clock_drift_allowed": "0.2",
> > "mon_clock_drift_warn_backoff": "5",
> > "clock_offset": "0",
> > root@ceph1:~#
> 
> Despite this setting, I keep receiving HEALTH_WARNs like below:
> 
> > ceph cluster node ceph1 health status became HEALTH_WARN clock skew 
> > detected on mon.1; Monitor clock skew detected mon.1 addr 10.10.89.2:6789/0 
> > clock skew 0.113709s > max 0.1s (latency 0.000523111s)
> 
> Can anyone explain why the running config shows 
> "mon_clock_drift_allowed": "0.2" and the HEALTH_WARN says "max 0.1s 
> (latency 0.000523111s)"?
> 
> How come there's a difference between the two?
> 
> MJ
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] clock skew

2017-04-01 Thread mj 

Hi!

On 04/01/2017 12:49 PM, Wei Jin wrote:

mon_clock_drift_allowed should be used in monitor process, what's the
output of `ceph daemon mon.foo config show | grep clock`?

how did you change the value? command line or config file?


I guess I changed it wrong then... Did it in ceph.conf, like:

[global]
mon clock drift allowed = 0.1

and for immediate effect, also:
> ceph tell osd.* injectargs --mon_clock_drift_allowed "0.2"

So I guess that's wrong..?

Should it be under the [mon] sections of ceph.conf?

If listed under [global] like I have it now, then what have does it 
actually change..?

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] clock skew

2017-04-01 Thread Wei Jin 
On Sat, Apr 1, 2017 at 5:17 PM, mj  wrote:
> Hi,
>
> Despite ntp, we keep getting clock skews that auto disappear again after a
> few minutes.
>
> To prevent the unneccerasy HEALTH_WARNs, I have increased the
> mon_clock_drift_allowed to 0.2, as can be seen below:
>
>> root@ceph1:~# ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config
>> show | grep clock
>> "mon_clock_drift_allowed": "0.2",
>> "mon_clock_drift_warn_backoff": "5",
>> "clock_offset": "0",
>> root@ceph1:~#

mon_clock_drift_allowed should be used in monitor process, what's the
output of `ceph daemon mon.foo config show | grep clock`?

how did you change the value? command line or config file?

>
>
> Despite this setting, I keep receiving HEALTH_WARNs like below:
>
>> ceph cluster node ceph1 health status became HEALTH_WARN clock skew
>> detected on mon.1; Monitor clock skew detected mon.1 addr 10.10.89.2:6789/0
>> clock skew 0.113709s > max 0.1s (latency 0.000523111s)
>
>
> Can anyone explain why the running config shows "mon_clock_drift_allowed":
> "0.2" and the HEALTH_WARN says "max 0.1s (latency 0.000523111s)"?
>
> How come there's a difference between the two?
>
> MJ
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Apply for an official mirror at CN

2017-04-01 Thread SJ Zhu 
Hi all,

We'd like to apply for the official mirror at China.
We have run the ceph mirror for more than one year, at
https://mirrors.ustc.edu.cn/ceph/

The server infomation is:
* located at USTC(University of Science and Technology of China)
* 2Gbit connection
* both IPv4/IPv6
* 60TB disk
* rsync service at rsync://rsync.mirrors.ustc.edu.cn/ceph/
* contact: lug(at)ustc.edu.cn

Thanks,
Shengjing Zhu

--
Linux User Group
University of Science and Technology of China
Homepage: https://lug.ustc.edu.cn/
E-Mail: l...@ustc.edu.cn
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] clock skew

2017-04-01 Thread mj 

Hi,

Despite ntp, we keep getting clock skews that auto disappear again after 
a few minutes.


To prevent the unneccerasy HEALTH_WARNs, I have increased the 
mon_clock_drift_allowed to 0.2, as can be seen below:



root@ceph1:~# ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config show | 
grep clock
"mon_clock_drift_allowed": "0.2",
"mon_clock_drift_warn_backoff": "5",
"clock_offset": "0",
root@ceph1:~#


Despite this setting, I keep receiving HEALTH_WARNs like below:


ceph cluster node ceph1 health status became HEALTH_WARN clock skew detected on 
mon.1; Monitor clock skew detected mon.1 addr 10.10.89.2:6789/0 clock skew 
0.113709s > max 0.1s (latency 0.000523111s)


Can anyone explain why the running config shows 
"mon_clock_drift_allowed": "0.2" and the HEALTH_WARN says "max 0.1s 
(latency 0.000523111s)"?


How come there's a difference between the two?

MJ
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] CentOS7 Mounting Problem

2017-04-01 Thread Georgios Dimitrakakis 


Hi,

just to provide some more feedback on this one and what I 've done to 
solve it, although not sure if this is the most "elegant" solution.


I have add manually to /etc/fstab on all systems the respective mount 
points for Ceph OSDs, e.g. entries like this:


UUID=9d2e7674-f143-48a2-bb7a-1c55b99da1f7 /var/lib/ceph/osd/ceph-0 xfs 
defaults		0 0



Then I 've checked and seen that the "ceph-osd@.service" was "disabled" 
which means that wasn't starting by default.


Therefore I did modify all respective services on all nodes, with 
commands like:


systemctl enable ceph-osd@0.service


Did a reboot on the nodes and all CEPH OSDs were mounted and the 
service was starting by default, so the problem was solved.


As said I don't know if this is the correct way to do it but for me it 
works.
I guess that something still goes wrong when the root volume is on LVM 
and all the above that should happen automatically don't happen and 
require manual intervention.


Looking forward for any comments on this procedure or things that I 
might have missed.


Regards,

G.



Hi Tom and thanks a lot for the feedback.

Indeed my root filesystem is on an LVM volume and I am currently
running CentOS 7.3.1611 with kernel 3.10.0-514.10.2.el7.x86_64 and 
the

ceph version is 10.2.6 (656b5b63ed7c43bd014bcafd81b001959d5f089f)

The 60-ceph-by-parttypeuuid.rules on the system is the same is the
one on the bug you 've mentioned but unfortunately it still doesn't
work.

So, are there any more ideas on how to further debug it?

Best,

G.



Are you running the CentOS filesystem as LVM? This
(http://tracker.ceph.com/issues/16351 [1]) still seems to be an 
issue

on CentOS 7 that I've seen myself too. After migrating to a standard
filesystem layout (i.e. no LVM) the issue disappeared.

Regards,

 Tom

-

FROM: ceph-users  on behalf of Georgios Dimitrakakis
 SENT: Thursday, March 23, 2017 10:21:34 PM
 TO: ceph-us...@ceph.com
 SUBJECT: [ceph-users] CentOS7 Mounting Problem

Hello Ceph community!

 I would like some help with a new CEPH installation.

 I have install Jewel on CentOS7 and after the reboot my OSDs are 
not

 mount automatically and as a consequence ceph is not operating
 normally...

 What can I do?

 Could you please help me solve the problem?

 Regards,

 G.
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com [2]


Links:
--
[1] http://tracker.ceph.com/issues/16351
[2] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] List-Archive unavailable

2017-04-01 Thread Robert Sander 
Hi,

the list archive at http://lists.ceph.com/pipermail/ceph-users-ceph.com/
is currently not available. Anybody knows what is going on there?

Regards
-- 
Robert Sander
Heinlein Support GmbH
Schwedter Str. 8/9b, 10119 Berlin

http://www.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Zwangsangaben lt. §35a GmbHG:
HRB 93818 B / Amtsgericht Berlin-Charlottenburg,
Geschäftsführer: Peer Heinlein -- Sitz: Berlin



signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com