Re: [ceph-users] VMware + CEPH Integration

[Adrian Saul]

> Hi Alex,
>
> Have you experienced any problems with timeouts in the monitor action in
> pacemaker? Although largely stable, every now and again in our cluster the
> FS and Exportfs resources timeout in pacemaker. There's no mention of any
> slow requests or any peering..etc from the ceph logs so it's a bit of a 
> mystery.

Yes - we have that in our setup which is very similar.  Usually  I find it 
related to RBD device latency  due to scrubbing or similar but even when tuning 
some of that down we still get it randomly.

The most annoying part is that once it comes up, having to use  "resource 
cleanup" to try and remove the failed usually has more impact than the actual 
error.
Confidentiality: This email and any attachments are confidential and may be 
subject to copyright, legal or some other professional privilege. They are 
intended solely for the attention and use of the named addressee(s). They may 
only be copied, distributed or disclosed with the consent of the copyright 
owner. If you have received this email by mistake or by breach of the 
confidentiality clause, please notify the sender immediately by return email 
and delete or destroy all copies of the email. Any confidentiality, privilege 
or copyright is not waived or lost because this email has been sent to you by 
mistake.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Mon Create currently at the state of probing

[Sasha Litvak]

Do you have firewall on on new server by any chance?

On Sun, Jun 18, 2017 at 8:18 PM, Jim Forde  wrote:

> I have an eight node ceph cluster running Jewel 10.2.5.
>
> One Ceph-Deploy node. Four OSD nodes and three Monitor nodes.
>
> Ceph-Deploy node is r710T
>
> OSD’s are r710a, r710b, r710c, and r710d.
>
> Mon’s are r710e, r710f, and r710g.
>
> Name resolution is in Hosts file on each node.
>
>
>
> Successfully removed Monitor r710e from cluster
>
> Upgraded ceph-deploy node r710T to Kraken 11.2.0 (ceph -v returns 11.2.0
> all other nodes are still 10.2.5)
>
> Ceph -s is HEALTH_OK 2 mons
>
> Rebuilt r710e with same OS (ubutnu 14.04 LTS) and same IP address.
>
> “Ceph-deploy install –release kraken r710e” is successful with ceph -v
> returning 11.2.0 on node r710e
>
> “ceph-deploy admin r710e” is successful and puts the keyring in
> /etc/ceph/ceph.client.admin.keyring
>
> “sudo chmod +r /etc/ceph/ceph.client.admin.keyring”
>
>
>
> Everything seems successful to this point.
>
> Then I run
>
> “ceph-deploy mon create r710e” and I get the following:
>
>
>
> [r710e][DEBUG ] **
> **
>
> [r710e][INFO  ] monitor: mon.r710e is currently at the state of probing
>
> [r710e][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon
> /var/run/ceph/ceph-mon.r710e.asok mon_status
>
> [r710e][WARNIN] r710e is not defined in `mon initial members`
>
> [r710e][WARNIN] monitor r710e does not exist in monmap
>
>
>
> R710e is in the ‘mon initial members’.
>
> It is in the ceph.conf file correctly (it was running before and the
> parameters have not changed) Public and Cluster networks are defined.
>
> It is the same physical server with the same (but freshly installed) OS
> and same IP address.
>
> Looking at the local daemon mon_status on all three monitors I see.
>
> R710f and r710g see r710e as an “extra_probe_peers”
>
> R710e sees r710f and r710g as “extra_probe_peers”
>
>
>
> “ceph-deploy purge r710e” and “ceph-deploy purgedata r710e” with a reboot
> of the 2 mon’s brings cluster back to HEALTH_OK
>
>
>
> Not sure what is going on. Is Ceph allergic to single node upgrades?
> Afraid to push the upgrade on all mon’s.
>
>
>
> What I have done:
>
> Rebuilt r710e with different hardware. Rebuilt with different OS. Rebuilt
> with different name and IP address. Same result.
>
> I have also restructured the NTP server. R710T is my NTP server on the
> cluster. (HEALTH_OK prior to updating) I reset all Mon nodes to get time
> from Ubuntu default NTP sources. Same error.
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Kernel RBD client talking to multiple storage clusters

[Alex Gorbachev]

Has anyone run into such config where a single client consumes storage from
several ceph clusters, unrelated to each other (different MONs and OSDs,
and keys)?

We have a Hammer and a Jewel cluster now, and this may be a way to have
very clean migrations.

Best regards,
Alex
Storcium
-- 
--
Alex Gorbachev
Storcium
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Mon Create currently at the state of probing

[Jim Forde]

I have an eight node ceph cluster running Jewel 10.2.5.
One Ceph-Deploy node. Four OSD nodes and three Monitor nodes.
Ceph-Deploy node is r710T
OSD's are r710a, r710b, r710c, and r710d.
Mon's are r710e, r710f, and r710g.
Name resolution is in Hosts file on each node.

Successfully removed Monitor r710e from cluster
Upgraded ceph-deploy node r710T to Kraken 11.2.0 (ceph -v returns 11.2.0 all 
other nodes are still 10.2.5)
Ceph -s is HEALTH_OK 2 mons
Rebuilt r710e with same OS (ubutnu 14.04 LTS) and same IP address.
"Ceph-deploy install -release kraken r710e" is successful with ceph -v 
returning 11.2.0 on node r710e
"ceph-deploy admin r710e" is successful and puts the keyring in 
/etc/ceph/ceph.client.admin.keyring
"sudo chmod +r /etc/ceph/ceph.client.admin.keyring"

Everything seems successful to this point.
Then I run
"ceph-deploy mon create r710e" and I get the following:

[r710e][DEBUG ] 

[r710e][INFO  ] monitor: mon.r710e is currently at the state of probing
[r710e][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon 
/var/run/ceph/ceph-mon.r710e.asok mon_status
[r710e][WARNIN] r710e is not defined in `mon initial members`
[r710e][WARNIN] monitor r710e does not exist in monmap

R710e is in the 'mon initial members'.
It is in the ceph.conf file correctly (it was running before and the parameters 
have not changed) Public and Cluster networks are defined.
It is the same physical server with the same (but freshly installed) OS and 
same IP address.
Looking at the local daemon mon_status on all three monitors I see.
R710f and r710g see r710e as an "extra_probe_peers"
R710e sees r710f and r710g as "extra_probe_peers"

"ceph-deploy purge r710e" and "ceph-deploy purgedata r710e" with a reboot of 
the 2 mon's brings cluster back to HEALTH_OK

Not sure what is going on. Is Ceph allergic to single node upgrades? Afraid to 
push the upgrade on all mon's.

What I have done:
Rebuilt r710e with different hardware. Rebuilt with different OS. Rebuilt with 
different name and IP address. Same result.
I have also restructured the NTP server. R710T is my NTP server on the cluster. 
(HEALTH_OK prior to updating) I reset all Mon nodes to get time from Ubuntu 
default NTP sources. Same error.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] What package I need to install to have CephFS kernel support on CentOS?

[John Spray]

On Fri, Jun 16, 2017 at 4:05 PM, Stéphane Klein
 wrote:
> Hi,
>
> I would like to use CephFS kernel module on CentOS 7
>
> I use Atomic version of CentOS.
>
> I don't know where is the CephFS kermel module rpm package.
>
> I have installed: libcephfs1-10.2.7-0.el7.x86_64 package and I have this:
>
> -bash-4.2# rpm -qvl libcephfs1
> lrwxrwxrwx1 rootroot   18 Apr 11 03:51
> /usr/lib64/libcephfs.so.1 -> libcephfs.so.1.0.0
> -rwxr-xr-x1 rootroot  6237376 Apr 11 04:32
> /usr/lib64/libcephfs.so.1.0.0
>
> I use this repos:
> https://raw.githubusercontent.com/CentOS-Storage-SIG/centos-release-ceph-jewel/master/CentOS-Ceph-Jewel.repo
>
> Question: what package I need to install to have CephFS kernel support on
> CentOS?

As David has already mentioned, if your kernel was built with CephFS
then no extra package will be needed.  We do include an extra
mount.ceph helper binary (it can e.g. handle mon DNS names instead of
IPs) in the ceph-common package, but it isn't mandatory.  Note that
not all distributions enable the cephfs kernel module in their builds,
although from a quick look on a centos box here it seems to be there.

However, please bear in mind that many distributions are shipping
kernels with seriously old cephfs clients, see
http://docs.ceph.com/docs/master/cephfs/best-practices/#which-kernel-version
-- if your distribution does not say anything about actively
supporting and backporting the cephfs client, then you can probably
assume that they aren't.  In this case, you may be better off
installing a much more recent kernel than the one that came with your
distribution, in order to get the latest cephfs code.

John

> Best regards,
> Stéphane
> --
> Stéphane Klein 
> blog: http://stephane-klein.info
> cv : http://cv.stephane-klein.info
> Twitter: http://twitter.com/klein_stephane
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD node type/count mixes in the cluster

[Mehmet]

Hi,

We actually using 3xIntel Server with 12 osds and One supermicro with 24 osds 
in One ceph Cluster Journals on nvme per server. Did not seeing any issues jet. 

Best
Mehmet

Am 9. Juni 2017 19:24:40 MESZ schrieb Deepak Naidu :
>Thanks David for sharing your experience, appreciate it.
>
>--
>Deepak
>
>From: David Turner [mailto:drakonst...@gmail.com]
>Sent: Friday, June 09, 2017 5:38 AM
>To: Deepak Naidu; ceph-users@lists.ceph.com
>Subject: Re: [ceph-users] OSD node type/count mixes in the cluster
>
>
>I ran a cluster with 2 generations of the same vendor hardware. 24 osd
>supermicro and 32 osd supermicro (with faster/more RAM and CPU cores). 
>The cluster itself ran decently well, but the load differences was
>drastic between the 2 types of nodes. It required me to run the cluster
>with 2 separate config files for each type of node and was an utter
>PITA when troubleshooting bottlenecks.
>
>Ultimately I moved around hardware and created a legacy cluster on the
>old hardware and created a new cluster using the newer configuration. 
>In general it was very hard to diagnose certain bottlenecks due to
>everything just looking so different.  The primary one I encountered
>was snap trimming due to deleted thousands of snapshots/day.
>
>If you aren't pushing any limits of Ceph, you will probably be fine. 
>But if you have a really large cluster, use a lot of snapshots, or are
>pushing your cluster harder than the average user... Then I'd avoid
>mixing server configurations in a cluster.
>
>On Fri, Jun 9, 2017, 1:36 AM Deepak Naidu
>> wrote:
>Wanted to check if anyone has a ceph cluster which has mixed vendor
>servers both with same disk size i.e. 8TB but different count i.e.
>Example 10 OSD servers from Dell with 60 Disk per server and other 10
>OSD servers from HP with 26 Disk per server.
>
>If so does that change any performance dynamics ? or is it not
>advisable .
>
>--
>Deepak
>---
>This email message is for the sole use of the intended recipient(s) and
>may contain
>confidential information.  Any unauthorized review, use, disclosure or
>distribution
>is prohibited.  If you are not the intended recipient, please contact
>the sender by
>reply email and destroy all copies of the original message.
>---
>___
>ceph-users mailing list
>ceph-users@lists.ceph.com
>http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] FAILED assert(i.first <= i.last)

[Peter Rosell]

Hi,
I have a small cluster with only three nodes, 4 OSDs + 3 OSDs. I have been
running version 0.87.2 (Giant) for over 2.5 year, but a couple of day ago I
upgraded to 0.94.10 (Hammer) and then up to 10.2.7 (Jewel). Both the
upgrades went great. Started with monitors, osd and finally mds. The log
shows all 448 pgs active+clean. I'm running all daemons inside docker and
ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)

Today I had a power outage and I had to take down the servers. When I now
start the servers again one OSD daemon doesn't start properly. It keeps
crashing.

I noticed that the two first restarts of the osd daemon crashed with this
error:
FAILED assert(rollback_info_trimmed_to_riter == log.rbegin())

After that it always fails with "FAILED assert(i.first <= i.last)"

I have 15 logs like this one:
Jun 18 08:56:18 island sh[27991]: 2017-06-18 08:56:18.300641 7f5c5e0ff8c0
-1 log_channel(cluster) log [ERR] : 2.38 log bound mismatch, info
(19544'666742,19691'671046] actual [19499'665843,19691'671046]

I removed the directories _head, but that just removed these error
logs. It crashes anyway.

Anyone has any suggestions what to do to make it start up correct. Of
course I can remove the OSD from the cluster and re-add it, but it feels
like a bug.
A small snippet from the logs is added below. I didn't include the event
list. If it will help I can send it too.

Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: In function 'static bool
pg_interval_t::check_new_interval(int, int, const std::vector&, const
std::vector&, int, int, const std::vector&, const
std::vector&, epoch_t, epoch_t, OSDMapRef, OSDMapRef, pg_t,
IsPGRecoverablePredicate*, std::map*,
std::ostream*)' thread 7f4fc2500700 time 2017-06-18 13:52:23.593991
Jun 18 13:52:23 island sh[7068]: osd/osd_types.cc: 3132: FAILED
assert(i.first <= i.last)
Jun 18 13:52:23 island sh[7068]:  ceph version 10.2.7
(50e863e0f4bc8f4b9e31156de690d765af245185)
Jun 18 13:52:23 island sh[7068]:  1: (ceph::__ceph_assert_fail(char const*,
char const*, int, char const*)+0x80) [0x559fe4c14360]
Jun 18 13:52:23 island sh[7068]:  2:
(pg_interval_t::check_new_interval(int, int, std::vector const&, std::vector
const&, int, int, std::vector const&,
std::vector const&, unsigned int, unsigned int,
std::shared_ptr, std::shared_ptr, pg_t,
IsPGRecoverablePredicate*, std::map, std::allocator >*, std::ostream*)+0x72c) [0x559fe47f723c]
Jun 18 13:52:23 island sh[7068]:  3:
(PG::start_peering_interval(std::shared_ptr, std::vector const&, int, std::vector
const&, int, ObjectStore::Transaction*)+0x3ff) [0x559fe461439f]
Jun 18 13:52:23 island sh[7068]:  4:
(PG::RecoveryState::Reset::react(PG::AdvMap const&)+0x478) [0x559fe4615828]
Jun 18 13:52:23 island sh[7068]:  5:
(boost::statechart::simple_state,
(boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base
const&, void const*)+0x176) [0x559fe4645b86]
Jun 18 13:52:23 island sh[7068]:  6:
(boost::statechart::state_machine::process_event(boost::statechart::event_base
const&)+0x69) [0x559fe4626d49]
Jun 18 13:52:23 island sh[7068]:  7:
(PG::handle_advance_map(std::shared_ptr,
std::shared_ptr, std::vector&,
int, std::vector&, int, PG::RecoveryCtx*)+0x49e)
[0x559fe45fa5ae]
Jun 18 13:52:23 island sh[7068]:  8: (OSD::advance_pg(unsigned int, PG*,
ThreadPool::TPHandle&, PG::RecoveryCtx*, std::set,
std::allocator >*)+0x2f2) [0x559fe452c042]
Jun 18 13:52:23 island sh[7068]:  9:
(OSD::process_peering_events(std::__cxx11::list >
const&, ThreadPool::TPHandle&)+0x214) [0x559fe4546d34]
Jun 18 13:52:23 island sh[7068]:  10:
(ThreadPool::BatchWorkQueue::_void_process(void*,
ThreadPool::TPHandle&)+0x25) [0x559fe458f8e5]
Jun 18 13:52:23 island sh[7068]:  11:
(ThreadPool::worker(ThreadPool::WorkThread*)+0xdb1) [0x559fe4c06531]
Jun 18 13:52:23 island sh[7068]:  12:
(ThreadPool::WorkThread::entry()+0x10) [0x559fe4c07630]
Jun 18 13:52:23 island sh[7068]:  13: (()+0x76fa) [0x7f4fe256b6fa]
Jun 18 13:52:23 island sh[7068]:  14: (clone()+0x6d) [0x7f4fe05e3b5d]
Jun 18 13:52:23 island sh[7068]:  NOTE: a copy of the executable, or
`objdump -rdS ` is needed to interpret this.
Jun 18 13:52:23 island sh[7068]: 2017-06-18 13:52:23.599558 7f4fc2500700 -1
osd/osd_types.cc: In function 'static bool
pg_interval_t::check_new_interval(int, int, const