Re: [ceph-users] Creating a monmap with V1 & V2 using monmaptool

[Lars Fenneberg]

Hey Alberto!

Quoting Corona, Alberto (alberto_cor...@comcast.com):

> While practicing some disaster recovery I noticed that it currently seems
> impossible to add both a v1 and v2 monitor to a monmap using monmaptool. 
> Is there any way to create a monmap manually to include both protocol
> versions to a monmap?  Currently on Ceph version 14.2.4

There is a new --addv option which accepts the new monitor address syntax.

Cheers,
LF.
-- 
Lars Fenneberg, l...@elemental.net
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD crashed during the fio test

[Brad Hubbard]

Removed ceph-de...@vger.kernel.org and added d...@ceph.io

On Tue, Oct 1, 2019 at 4:26 PM Alex Litvak  wrote:
>
> Hellow everyone,
>
> Can you shed the line on the cause of the crash?  Could actually client 
> request trigger it?
>
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 22:52:58.867 
> 7f093d71e700 -1 bdev(0x55b72c156000 /var/lib/ceph/osd/ceph-17/block) 
> aio_submit retries 16
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 22:52:58.867 
> 7f093d71e700 -1 bdev(0x55b72c156000 /var/lib/ceph/osd/ceph-17/block)  aio 
> submit got (11) Resource temporarily unavailable

The KernelDevice::aio_submit function has tried to submit Io 16 times
(a hard coded limit) and received an error each time causing it to
assert. Can you check the status of the underlying device(s)?

> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc:
> In fun
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc:
> 757: F
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  ceph version 14.2.2 
> (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  1: 
> (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) 
> [0x55b71f668cf4]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  2: 
> (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char 
> const*, ...)+0) [0x55b71f668ec2]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  3: 
> (KernelDevice::aio_submit(IOContext*)+0x701) [0x55b71fd61ca1]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  4: 
> (BlueStore::_txc_aio_submit(BlueStore::TransContext*)+0x42) [0x55b71fc29892]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  5: 
> (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x42b) [0x55b71fc496ab]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  6: 
> (BlueStore::queue_transactions(boost::intrusive_ptr&,
>  std::vector std::allocator >&, boost::intrusive_ptr, 
> ThreadPool::T
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  7: (non-virtual thunk to 
> PrimaryLogPG::queue_transactions(std::vector std::allocator >&,
> boost::intrusive_ptr)+0x54) [0x55b71f9b1b84]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  8: 
> (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t 
> const&, eversion_t const&, std::unique_ptr std::default_delete >&&, eversion_t const&, eversion_t const&, 
> s
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  9: 
> (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, 
> PrimaryLogPG::OpContext*)+0xf12) [0x55b71f90e322]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  10: 
> (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfae) [0x55b71f969b7e]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  11: 
> (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3965) 
> [0x55b71f96de15]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  12: 
> (PrimaryLogPG::do_request(boost::intrusive_ptr&, 
> ThreadPool::TPHandle&)+0xbd4) [0x55b71f96f8a4]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  13: 
> (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, 
> ThreadPool::TPHandle&)+0x1a9) [0x55b71f7a9ea9]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  14: (PGOpItem::run(OSD*, 
> OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x62) 
> [0x55b71fa475d2]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  15: 
> (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x9f4) 
> [0x55b71f7c6ef4]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  16: 
> (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x433) 
> [0x55b71fdc5ce3]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  17: 
> (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55b71fdc8d80]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  18: (()+0x7dd5) 
> [0x7f0971da9dd5]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  19: (clone()+0x6d) 
> [0x7f0970c7002d]
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 22:52:58.879 
> 7f093d71e700 -1
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc:
> 757: F
> Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  ceph version 

[ceph-users] OSD crashed during the fio test

[Alex Litvak]

Hellow everyone,

Can you shed the line on the cause of the crash?  Could actually client request 
trigger it?

Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 22:52:58.867 
7f093d71e700 -1 bdev(0x55b72c156000 /var/lib/ceph/osd/ceph-17/block) aio_submit 
retries 16
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 22:52:58.867 
7f093d71e700 -1 bdev(0x55b72c156000 /var/lib/ceph/osd/ceph-17/block)  aio 
submit got (11) Resource temporarily unavailable
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc: 
In fun
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc: 
757: F

Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  ceph version 14.2.2 
(4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  1: 
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) 
[0x55b71f668cf4]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  2: 
(ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char 
const*, ...)+0) [0x55b71f668ec2]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  3: 
(KernelDevice::aio_submit(IOContext*)+0x701) [0x55b71fd61ca1]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  4: 
(BlueStore::_txc_aio_submit(BlueStore::TransContext*)+0x42) [0x55b71fc29892]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  5: 
(BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x42b) [0x55b71fc496ab]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  6: (BlueStore::queue_transactions(boost::intrusive_ptr&, std::vectorstd::allocator >&, boost::intrusive_ptr, ThreadPool::T
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  7: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, 
boost::intrusive_ptr)+0x54) [0x55b71f9b1b84]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  8: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptrstd::default_delete >&&, eversion_t const&, eversion_t const&, s

Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  9: 
(PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, 
PrimaryLogPG::OpContext*)+0xf12) [0x55b71f90e322]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  10: 
(PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfae) [0x55b71f969b7e]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  11: 
(PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3965) [0x55b71f96de15]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  12: 
(PrimaryLogPG::do_request(boost::intrusive_ptr&, 
ThreadPool::TPHandle&)+0xbd4) [0x55b71f96f8a4]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  13: 
(OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, 
ThreadPool::TPHandle&)+0x1a9) [0x55b71f7a9ea9]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  14: (PGOpItem::run(OSD*, OSDShard*, 
boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x62) [0x55b71fa475d2]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  15: 
(OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x9f4) 
[0x55b71f7c6ef4]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  16: 
(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x433) 
[0x55b71fdc5ce3]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  17: 
(ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55b71fdc8d80]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  18: (()+0x7dd5) 
[0x7f0971da9dd5]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  19: (clone()+0x6d) 
[0x7f0970c7002d]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 22:52:58.879 7f093d71e700 -1 
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc: 
757: F

Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  ceph version 14.2.2 
(4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  1: 
(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) 
[0x55b71f668cf4]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  2: 
(ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char 
const*, ...)+0) [0x55b71f668ec2]
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  3: 
(KernelDevice::aio_submit(IOContext*)+0x701) 

Re: [ceph-users] cluster network down

[Lars Täuber]

Mon, 30 Sep 2019 15:21:18 +0200
Janne Johansson  ==> Lars Täuber  :
> >
> > I don't remember where I read it, but it was told that the cluster is
> > migrating its complete traffic over to the public network when the cluster
> > networks goes down. So this seems not to be the case?
> >  
> 
> Be careful with generalizations like "when a network acts up, it will be
> completely down and noticeably unreachable for all parts", since networks
> can break in thousands of not-very-obvious ways which are not 0%-vs-100%
> but somewhere in between.
> 

Ok. I ask my question in a new way.
What does ceph do, when I switch off all switches of the cluster network?
Does ceph handle this silently without interruption? Does the heartbeat systems 
use the public network as a failover automatically?

Thanks
Lars
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph-osd@n crash dumps

[Del Monaco, Andrea]

Hi list,

After the nodes ran OOM and after reboot, we are not able to restart the 
ceph-osd@x services anymore. (Details about the setup at the end).

I am trying to do this manually, so we can see the error but all i see is 
several crash dumps - this is just one of the OSDs which is not starting. Any 
idea how to get past this??
[root@ceph001 ~]# /usr/bin/ceph-osd --debug_osd 10 -f --cluster ceph --id 83 
--setuser ceph --setgroup ceph  > /tmp/dump 2>&1
starting osd.83 at - osd_data /var/lib/ceph/osd/ceph-83 
/var/lib/ceph/osd/ceph-83/journal
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h:
 In function 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)' thread 
2aaf5540 time 2019-10-01 14:19:49.494368
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h:
 34: FAILED assert(stripe_width % stripe_size == 0)
 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x14b) [0x2af3d36b]
 2: (()+0x26e4f7) [0x2af3d4f7]
 3: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, 
boost::intrusive_ptr&, ObjectStore*, CephContext*, 
std::shared_ptr, unsigned long)+0x46d) 
[0x55c0bd3d]
 4: (PGBackend::build_pg_backend(pg_pool_t const&, std::map, std::allocator > > const&, PGBackend::Listener*, coll_t, 
boost::intrusive_ptr&, ObjectStore*, 
CephContext*)+0x30a) [0x55b0ba8a]
 5: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr, 
PGPool const&, std::map, 
std::allocator > > const&, 
spg_t)+0x140) [0x55abd100]
 6: (OSD::_make_pg(std::shared_ptr, spg_t)+0x10cb) 
[0x55914ecb]
 7: (OSD::load_pgs()+0x4a9) [0x55917e39]
 8: (OSD::init()+0xc99) [0x559238e9]
 9: (main()+0x23a3) [0x558017a3]
 10: (__libc_start_main()+0xf5) [0x2aaab77de495]
 11: (()+0x385900) [0x558d9900]
2019-10-01 14:19:49.500 2aaf5540 -1 
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h:
 In function 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)' thread 
2aaf5540 time 2019-10-01 14:19:49.494368
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h:
 34: FAILED assert(stripe_width % stripe_size == 0)

 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x14b) [0x2af3d36b]
 2: (()+0x26e4f7) [0x2af3d4f7]
 3: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, 
boost::intrusive_ptr&, ObjectStore*, CephContext*, 
std::shared_ptr, unsigned long)+0x46d) 
[0x55c0bd3d]
 4: (PGBackend::build_pg_backend(pg_pool_t const&, std::map, std::allocator > > const&, PGBackend::Listener*, coll_t, 
boost::intrusive_ptr&, ObjectStore*, 
CephContext*)+0x30a) [0x55b0ba8a]
 5: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr, 
PGPool const&, std::map, 
std::allocator > > const&, 
spg_t)+0x140) [0x55abd100]
 6: (OSD::_make_pg(std::shared_ptr, spg_t)+0x10cb) 
[0x55914ecb]
 7: (OSD::load_pgs()+0x4a9) [0x55917e39]
 8: (OSD::init()+0xc99) [0x559238e9]
 9: (main()+0x23a3) [0x558017a3]
 10: (__libc_start_main()+0xf5) [0x2aaab77de495]
 11: (()+0x385900) [0x558d9900]

*** Caught signal (Aborted) **
 in thread 2aaf5540 thread_name:ceph-osd
 ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic (stable)
 1: (()+0xf5d0) [0x2aaab69765d0]
 2: (gsignal()+0x37) [0x2aaab77f22c7]
 3: (abort()+0x148) [0x2aaab77f39b8]
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char 
const*)+0x248) [0x2af3d468]
 5: (()+0x26e4f7) [0x2af3d4f7]
 6: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&, 
boost::intrusive_ptr&, ObjectStore*, CephContext*, 
std::shared_ptr, unsigned long)+0x46d) 
[0x55c0bd3d]
 7: (PGBackend::build_pg_backend(pg_pool_t const&, std::map, std::allocator > > const&, PGBackend::Listener*, coll_t, 
boost::intrusive_ptr&, ObjectStore*, 
CephContext*)+0x30a) [0x55b0ba8a]
 8: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr, 
PGPool const&, std::map, 
std::allocator > > const&, 
spg_t)+0x140) [0x55abd100]
 9: (OSD::_make_pg(std::shared_ptr, spg_t)+0x10cb) 
[0x55914ecb]
 10: (OSD::load_pgs()+0x4a9) [0x55917e39]
 11: (OSD::init()+0xc99) [0x559238e9]
 12: (main()+0x23a3) [0x558017a3]
 13: (__libc_start_main()+0xf5) [0x2aaab77de495]
 14: (()+0x385900) [0x558d9900]
2019-10-01 14:19:49.509 2aaf5540 -1 *** Caught signal (Aborted) 

Re: [ceph-users] NFS

[Marc Roos]

Yes indeed cephfs and rgw backends. I think you can run into problems 
with a multi user environment of RGW and nfs-ganesha. I am not getting 
this working on Luminous. Your rgw config seems ok. 

Add file logging to debug rgw etc something like this.


LOG {
## Default log level for all components
# NULL, FATAL, MAJ, CRIT, WARN, EVENT, INFO, DEBUG, MID_DEBUG, 
M_DBG, FULL_DEBUG, F_DBG], default EVENT
default_log_level = INFO;
#default_log_level = DEBUG;

## Configure per-component log levels.
# ALL, 
LOG,LOG_EMERG,MEMLEAKS,FSAL,NFSPROTO(NFS3),NFS_V4(NFSV4),EXPORT,FILEHAND
LE,DISPATCH,CACHE_INODE,
# CACHE_INODE_LRU,HASHTABLE,HASHTABLE_CACHE,DUPREQ,INIT,MAIN, 
IDMAPPER,NFS_READDIR,NFS_V4_LOCK,CONFIG,CLIENTID,
# 
SESSIONS,PNFS,RW_LOCK,NLM,RPC,NFS_CB,THREAD,NFS_V4_ACL,STATE,9P,9P_DISPA
TCH,FSAL_UP,DBUS

Components {
ALL = WARN;
#ALL = DEBUG;
#FSAL = F_DBG;
#NFS4 = F_DBG;
#EXPORT = F_DBG;
#CONFIG = F_DBG;
}

## Where to log
#   Facility {
#   name = FILE;
#   destination = "/var/log/ganesha.log";
#   enable = default;
#   }
}

-Original Message-
Subject: Re: [ceph-users] NFS

Ganesha can export CephFS or RGW.  It cannot export anything else (like 
iscsi or RBD).  Config for RGW looks like this:

EXPORT
{
 Export_ID=1;
 Path = "/";
 Pseudo = "/rgw";
 Access_Type = RW;
 Protocols = 4;
 Transports = TCP;
 FSAL {
 Name = RGW;
 User_Id = "testuser";
 Access_Key_Id ="";
 Secret_Access_Key = "";
 }
}

RGW {
 ceph_conf = "//ceph.conf";
 # for vstart cluster, name = "client.admin"
 name = "client.rgw.foohost";
 cluster = "ceph";
#   init_args = "-d --debug-rgw=16";
}


Daniel

On 9/30/19 3:01 PM, Marc Roos wrote:
>   
> Just install these
> 
> http://download.ceph.com/nfs-ganesha/
> nfs-ganesha-rgw-2.7.1-0.1.el7.x86_64
> nfs-ganesha-vfs-2.7.1-0.1.el7.x86_64
> libnfsidmap-0.25-19.el7.x86_64
> nfs-ganesha-mem-2.7.1-0.1.el7.x86_64
> nfs-ganesha-xfs-2.7.1-0.1.el7.x86_64
> nfs-ganesha-2.7.1-0.1.el7.x86_64
> nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64
> 
> 
> And export your cephfs like this:
> EXPORT {
>  Export_Id = 10;
>  Path = /nfs/cblr-repos;
>  Pseudo = /cblr-repos;
>  FSAL { Name = CEPH; User_Id = "cephfs.nfs.cblr"; 
> Secret_Access_Key = "xxx"; }
>  Disable_ACL = FALSE;
>  CLIENT { Clients = 192.168.10.2; access_type = "RW"; }
>  CLIENT { Clients = 192.168.10.253; } }
> 
> 
> -Original Message-
> From: Brent Kennedy [mailto:bkenn...@cfl.rr.com]
> Sent: maandag 30 september 2019 20:56
> To: 'ceph-users'
> Subject: [ceph-users] NFS
> 
> Wondering if there are any documents for standing up NFS with an 
> existing ceph cluster.  We don’t use ceph-ansible or any other tools 
> besides ceph-deploy.  The iscsi directions were pretty good once I got 

> past the dependencies.
> 
>   
> 
> I saw the one based on Rook, but it doesn’t seem to apply to our 
setup 
> of ceph vms with physical hosts doing OSDs.  The official ceph 
> documents talk about using ganesha but doesn’t seem to dive into the 
> details of what the process is for getting it online.  We don’t use 
> cephfs, so that’s not setup either.  The basic docs seem to note this 
is required.
>   Seems my google-fu is failing me when I try to find a more 
> definitive guide.
> 
>   
> 
> The servers are all centos 7 with the latest updates.
> 
>   
> 
> Any guidance would be greatly appreciated!
> 
>   
> 
> Regards,
> 
> -Brent
> 
>   
> 
> Existing Clusters:
> 
> Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 
> iscsi gateways ( all virtual on nvme )
> 
> US Production(HDD): Nautilus 14.2.2 with 13 osd servers, 3 mons, 4 
> gateways, 2 iscsi gateways
> 
> UK Production(HDD): Nautilus 14.2.2 with 25 osd servers, 3 mons/man, 3 

> gateways behind
> 
> US Production(SSD): Nautilus 14.2.2 with 6 osd servers, 3 mons/man, 3 
> gateways, 2 iscsi gateways
> 
>   
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] ceph pg repair fails...?

[Jake Grimmett]

Dear All,

I've just found two inconsistent pg that fail to repair.

This might be the same bug as shown here:



Cluster is running Nautilus 14.2.2
OS is Scientific Linux 7.6
DB/WAL on NVMe, Data on 12TB HDD

Logs below cab also be seen here: 

[root@ceph-s1 ~]# ceph health detail
HEALTH_ERR 22 scrub errors; Possible data damage: 2 pgs inconsistent
OSD_SCRUB_ERRORS 22 scrub errors
PG_DAMAGED Possible data damage: 2 pgs inconsistent
pg 2.2a7 is active+clean+inconsistent+failed_repair, acting
[83,60,133,326,281,162,180,172,144,219]
pg 2.36b is active+clean+inconsistent+failed_repair, acting
[254,268,10,262,32,280,211,114,169,53]

Issued "pg repair" commands, osd log shows:
[root@ceph-n10 ~]# grep "2.2a7" /var/log/ceph/ceph-osd.83.log
2019-10-01 07:05:02.459 7f9adab4b700  0 log_channel(cluster) log [DBG] :
2.2a7 repair starts
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 83(0) soid 2:e5472cab:::1000702081f.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 60(1) soid 2:e5472cab:::1000702081f.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 133(2) soid 2:e5472cab:::1000702081f.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 144(8) soid 2:e5472cab:::1000702081f.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 162(5) soid 2:e5472cab:::1000702081f.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 172(7) soid 2:e5472cab:::1000702081f.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 180(6) soid 2:e5472cab:::1000702081f.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 219(9) soid 2:e5472cab:::1000702081f.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 281(4) soid 2:e5472cab:::1000702081f.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 shard 326(3) soid 2:e5472cab:::1000702081f.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 soid 2:e5472cab:::1000702081f.:head : failed to pick
suitable object info
2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
repair 2.2a7s0 2:e5472cab:::1000702081f.:head : on disk size
(4096) does not match object info size (0) adjusted for ondisk to (0)
2019-10-01 07:19:47.060 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
2.2a7 repair 11 errors, 0 fixed
[root@ceph-n10 ~]#

[root@ceph-s1 ~]#  ceph pg repair 2.36b
instructing pg 2.36bs0 on osd.254 to repair

[root@ceph-n29 ~]# grep "2.36b" /var/log/ceph/ceph-osd.254.log
2019-10-01 11:15:12.215 7fa01f589700  0 log_channel(cluster) log [DBG] :
2.36b repair starts
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 254(0) soid 2:d6cac754:::100070209f6.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 10(2) soid 2:d6cac754:::100070209f6.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 32(4) soid 2:d6cac754:::100070209f6.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 53(9) soid 2:d6cac754:::100070209f6.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 114(7) soid 2:d6cac754:::100070209f6.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 169(8) soid 2:d6cac754:::100070209f6.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 211(6) soid 2:d6cac754:::100070209f6.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 262(3) soid 2:d6cac754:::100070209f6.:head :
candidate size 4096 info size 0 mismatch
2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
2.36b shard 268(1) soid 

Re: [ceph-users] NFS

[Daniel Gryniewicz]

Ganesha can export CephFS or RGW.  It cannot export anything else (like 
iscsi or RBD).  Config for RGW looks like this:


EXPORT
{
Export_ID=1;
Path = "/";
Pseudo = "/rgw";
Access_Type = RW;
Protocols = 4;
Transports = TCP;
FSAL {
Name = RGW;
User_Id = "testuser";
Access_Key_Id ="";
Secret_Access_Key = "";
}
}

RGW {
ceph_conf = "//ceph.conf";
# for vstart cluster, name = "client.admin"
name = "client.rgw.foohost";
cluster = "ceph";
#   init_args = "-d --debug-rgw=16";
}


Daniel

On 9/30/19 3:01 PM, Marc Roos wrote:
  
Just install these


http://download.ceph.com/nfs-ganesha/
nfs-ganesha-rgw-2.7.1-0.1.el7.x86_64
nfs-ganesha-vfs-2.7.1-0.1.el7.x86_64
libnfsidmap-0.25-19.el7.x86_64
nfs-ganesha-mem-2.7.1-0.1.el7.x86_64
nfs-ganesha-xfs-2.7.1-0.1.el7.x86_64
nfs-ganesha-2.7.1-0.1.el7.x86_64
nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64


And export your cephfs like this:
EXPORT {
 Export_Id = 10;
 Path = /nfs/cblr-repos;
 Pseudo = /cblr-repos;
 FSAL { Name = CEPH; User_Id = "cephfs.nfs.cblr";
Secret_Access_Key = "xxx"; }
 Disable_ACL = FALSE;
 CLIENT { Clients = 192.168.10.2; access_type = "RW"; }
 CLIENT { Clients = 192.168.10.253; }
}


-Original Message-
From: Brent Kennedy [mailto:bkenn...@cfl.rr.com]
Sent: maandag 30 september 2019 20:56
To: 'ceph-users'
Subject: [ceph-users] NFS

Wondering if there are any documents for standing up NFS with an
existing ceph cluster.  We don’t use ceph-ansible or any other tools
besides ceph-deploy.  The iscsi directions were pretty good once I got
past the dependencies.

  


I saw the one based on Rook, but it doesn’t seem to apply to our setup
of ceph vms with physical hosts doing OSDs.  The official ceph documents
talk about using ganesha but doesn’t seem to dive into the details of
what the process is for getting it online.  We don’t use cephfs, so
that’s not setup either.  The basic docs seem to note this is required.
  Seems my google-fu is failing me when I try to find a more definitive
guide.

  


The servers are all centos 7 with the latest updates.

  


Any guidance would be greatly appreciated!

  


Regards,

-Brent

  


Existing Clusters:

Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 iscsi
gateways ( all virtual on nvme )

US Production(HDD): Nautilus 14.2.2 with 13 osd servers, 3 mons, 4
gateways, 2 iscsi gateways

UK Production(HDD): Nautilus 14.2.2 with 25 osd servers, 3 mons/man, 3
gateways behind

US Production(SSD): Nautilus 14.2.2 with 6 osd servers, 3 mons/man, 3
gateways, 2 iscsi gateways

  



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD crashed during the fio test

[Sasha Litvak]

It was hardware indeed.  Dell server reported a disk being reset with power
on.  Checking the usual suspects i.e. controller firmware, controller event
log (if I can get one), drive firmware.
I will report more when I get a better idea

Thank you!

On Tue, Oct 1, 2019 at 2:33 AM Brad Hubbard  wrote:

> Removed ceph-de...@vger.kernel.org and added d...@ceph.io
>
> On Tue, Oct 1, 2019 at 4:26 PM Alex Litvak 
> wrote:
> >
> > Hellow everyone,
> >
> > Can you shed the line on the cause of the crash?  Could actually client
> request trigger it?
> >
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30
> 22:52:58.867 7f093d71e700 -1 bdev(0x55b72c156000
> /var/lib/ceph/osd/ceph-17/block) aio_submit retries 16
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30
> 22:52:58.867 7f093d71e700 -1 bdev(0x55b72c156000
> /var/lib/ceph/osd/ceph-17/block)  aio submit got (11) Resource temporarily
> unavailable
>
> The KernelDevice::aio_submit function has tried to submit Io 16 times
> (a hard coded limit) and received an error each time causing it to
> assert. Can you check the status of the underlying device(s)?
>
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:
> >
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc:
> > In fun
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:
> >
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc:
> > 757: F
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  ceph version 14.2.2
> (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  1:
> (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x14a) [0x55b71f668cf4]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  2:
> (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char
> const*, ...)+0) [0x55b71f668ec2]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  3:
> (KernelDevice::aio_submit(IOContext*)+0x701) [0x55b71fd61ca1]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  4:
> (BlueStore::_txc_aio_submit(BlueStore::TransContext*)+0x42) [0x55b71fc29892]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  5:
> (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x42b)
> [0x55b71fc496ab]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  6:
> (BlueStore::queue_transactions(boost::intrusive_ptr&,
> std::vector > std::allocator >&,
> boost::intrusive_ptr, ThreadPool::T
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  7: (non-virtual thunk
> to PrimaryLogPG::queue_transactions(std::vector std::allocator >&,
> > boost::intrusive_ptr)+0x54) [0x55b71f9b1b84]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  8:
> (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t
> const&, eversion_t const&, std::unique_ptr > std::default_delete >&&, eversion_t const&, eversion_t
> const&, s
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  9:
> (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*,
> PrimaryLogPG::OpContext*)+0xf12) [0x55b71f90e322]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  10:
> (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfae) [0x55b71f969b7e]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  11:
> (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3965)
> [0x55b71f96de15]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  12:
> (PrimaryLogPG::do_request(boost::intrusive_ptr&,
> ThreadPool::TPHandle&)+0xbd4) [0x55b71f96f8a4]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  13:
> (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr,
> ThreadPool::TPHandle&)+0x1a9) [0x55b71f7a9ea9]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  14:
> (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&,
> ThreadPool::TPHandle&)+0x62) [0x55b71fa475d2]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  15:
> (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x9f4)
> [0x55b71f7c6ef4]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  16:
> (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x433)
> [0x55b71fdc5ce3]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  17:
> (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55b71fdc8d80]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  18: (()+0x7dd5)
> [0x7f0971da9dd5]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  19: (clone()+0x6d)
> [0x7f0970c7002d]
> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30
> 22:52:58.879 7f093d71e700 -1
> >
> 

Re: [ceph-users] Have you enabled the telemetry module yet?

[Stefan Kooman]

Quoting Wido den Hollander (w...@42on.com):
> Hi,
> 
> The Telemetry [0] module has been in Ceph since the Mimic release and
> when enabled it sends back a anonymized JSON back to
> https://telemetry.ceph.com/ every 72 hours with information about the
> cluster.
> 
> For example:
> 
> - Version(s)
> - Number of MONs, OSDs, FS, RGW
> - Operating System used
> - CPUs used by MON and OSD
> 
> Enabling the module is very simple:
> 
> $ ceph mgr module enable telemetry

This worked.

ceph mgr module ls
{
"enabled_modules": [
...
...
"telemetry"
],

> Before enabling the module you can also view the JSON document it will
> send back:
> 
> $ ceph telemetry show

This gives me:

ceph telemetry show
Error EINVAL: Traceback (most recent call last):
  File "/usr/lib/ceph/mgr/telemetry/module.py", line 325, in handle_command
report = self.compile_report()
  File "/usr/lib/ceph/mgr/telemetry/module.py", line 291, in compile_report
report['crashes'] = self.gather_crashinfo()
  File "/usr/lib/ceph/mgr/telemetry/module.py", line 214, in gather_crashinfo
errno, crashids, err = self.remote('crash', 'do_ls', '', '')
  File "/usr/lib/ceph/mgr/mgr_module.py", line 845, in remote
args, kwargs)
ImportError: Module not found

Running 13.2.6 on Ubuntu Xenial 16.04.6 LTS

Gr. Stefan

-- 
| BIT BV  https://www.bit.nl/Kamer van Koophandel 09090351
| GPG: 0xD14839C6   +31 318 648 688 / i...@bit.nl
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Commit and Apply latency on nautilus

[Robert LeBlanc]

On Mon, Sep 30, 2019 at 5:12 PM Sasha Litvak
 wrote:
>
> At this point, I ran out of ideas.  I changed nr_requests and readahead 
> parameters to 128->1024 and 128->4096, tuned nodes to performance-throughput. 
>  However, I still get high latency during benchmark testing.  I attempted to 
> disable cache on ssd
>
> for i in {a..f}; do hdparm -W 0 -A 0 /dev/sd$i; done
>
> and I think it make things not better at all.  I have H740 and H730 
> controllers with drives in HBA mode.
>
> Other them converting them one by one to RAID0 I am not sure what else I can 
> try.
>
> Any suggestions?

If you haven't already tried this, add this to your ceph.conf and
restart your OSDs, this should help bring down the variance in latency
(It will be the default in Octopus):

osd op queue = wpq
osd op queue cut off = high


Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Commit and Apply latency on nautilus

[Robert LeBlanc]

On Tue, Oct 1, 2019 at 7:54 AM Robert LeBlanc  wrote:
>
> On Mon, Sep 30, 2019 at 5:12 PM Sasha Litvak
>  wrote:
> >
> > At this point, I ran out of ideas.  I changed nr_requests and readahead 
> > parameters to 128->1024 and 128->4096, tuned nodes to 
> > performance-throughput.  However, I still get high latency during benchmark 
> > testing.  I attempted to disable cache on ssd
> >
> > for i in {a..f}; do hdparm -W 0 -A 0 /dev/sd$i; done
> >
> > and I think it make things not better at all.  I have H740 and H730 
> > controllers with drives in HBA mode.
> >
> > Other them converting them one by one to RAID0 I am not sure what else I 
> > can try.
> >
> > Any suggestions?
>
> If you haven't already tried this, add this to your ceph.conf and
> restart your OSDs, this should help bring down the variance in latency
> (It will be the default in Octopus):
>
> osd op queue = wpq
> osd op queue cut off = high

I should clarify. This will reduce the variance in latency for client
OPs. If this counter is also including recovery/backfill/deep_scrub
OP-, then the latency can still be high as these settings make
recovery/backfill/deep_scrub less impactful to client I/O at the cost
of them possibly being delayed a bit.

Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Have you enabled the telemetry module yet?

[Mattia Belluco]

Hi all,

Same situation here:

Ceph 13.2.6 on Ubuntu 16.04.

Best
Mattia

On 10/1/19 4:38 PM, Stefan Kooman wrote:
> Quoting Wido den Hollander (w...@42on.com):
>> Hi,
>>
>> The Telemetry [0] module has been in Ceph since the Mimic release and
>> when enabled it sends back a anonymized JSON back to
>> https://telemetry.ceph.com/ every 72 hours with information about the
>> cluster.
>>
>> For example:
>>
>> - Version(s)
>> - Number of MONs, OSDs, FS, RGW
>> - Operating System used
>> - CPUs used by MON and OSD
>>
>> Enabling the module is very simple:
>>
>> $ ceph mgr module enable telemetry
> 
> This worked.
> 
> ceph mgr module ls
> {
> "enabled_modules": [
> ...
> ...
> "telemetry"
> ],
> 
>> Before enabling the module you can also view the JSON document it will
>> send back:
>>
>> $ ceph telemetry show
> 
> This gives me:
> 
> ceph telemetry show
> Error EINVAL: Traceback (most recent call last):
>   File "/usr/lib/ceph/mgr/telemetry/module.py", line 325, in handle_command
> report = self.compile_report()
>   File "/usr/lib/ceph/mgr/telemetry/module.py", line 291, in compile_report
> report['crashes'] = self.gather_crashinfo()
>   File "/usr/lib/ceph/mgr/telemetry/module.py", line 214, in gather_crashinfo
> errno, crashids, err = self.remote('crash', 'do_ls', '', '')
>   File "/usr/lib/ceph/mgr/mgr_module.py", line 845, in remote
> args, kwargs)
> ImportError: Module not found
> 
> Running 13.2.6 on Ubuntu Xenial 16.04.6 LTS
> 
> Gr. Stefan
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph pg repair fails...?

[Mattia Belluco]

Hi Jake,

I am curious to see if your problem is similar to ours (despite the fact
we are still on Luminous).

Could you post the output of:

rados list-inconsistent-obj 

and

rados list-inconsistent-snapset 

Thanks,

Mattia

On 10/1/19 1:08 PM, Jake Grimmett wrote:
> Dear All,
> 
> I've just found two inconsistent pg that fail to repair.
> 
> This might be the same bug as shown here:
> 
> 
> 
> Cluster is running Nautilus 14.2.2
> OS is Scientific Linux 7.6
> DB/WAL on NVMe, Data on 12TB HDD
> 
> Logs below cab also be seen here: 
> 
> [root@ceph-s1 ~]# ceph health detail
> HEALTH_ERR 22 scrub errors; Possible data damage: 2 pgs inconsistent
> OSD_SCRUB_ERRORS 22 scrub errors
> PG_DAMAGED Possible data damage: 2 pgs inconsistent
> pg 2.2a7 is active+clean+inconsistent+failed_repair, acting
> [83,60,133,326,281,162,180,172,144,219]
> pg 2.36b is active+clean+inconsistent+failed_repair, acting
> [254,268,10,262,32,280,211,114,169,53]
> 
> Issued "pg repair" commands, osd log shows:
> [root@ceph-n10 ~]# grep "2.2a7" /var/log/ceph/ceph-osd.83.log
> 2019-10-01 07:05:02.459 7f9adab4b700  0 log_channel(cluster) log [DBG] :
> 2.2a7 repair starts
> 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> 2.2a7 shard 83(0) soid 2:e5472cab:::1000702081f.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> 2.2a7 shard 60(1) soid 2:e5472cab:::1000702081f.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> 2.2a7 shard 133(2) soid 2:e5472cab:::1000702081f.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> 2.2a7 shard 144(8) soid 2:e5472cab:::1000702081f.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> 2.2a7 shard 162(5) soid 2:e5472cab:::1000702081f.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> 2.2a7 shard 172(7) soid 2:e5472cab:::1000702081f.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> 2.2a7 shard 180(6) soid 2:e5472cab:::1000702081f.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> 2.2a7 shard 219(9) soid 2:e5472cab:::1000702081f.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> 2.2a7 shard 281(4) soid 2:e5472cab:::1000702081f.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> 2.2a7 shard 326(3) soid 2:e5472cab:::1000702081f.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> 2.2a7 soid 2:e5472cab:::1000702081f.:head : failed to pick
> suitable object info
> 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> repair 2.2a7s0 2:e5472cab:::1000702081f.:head : on disk size
> (4096) does not match object info size (0) adjusted for ondisk to (0)
> 2019-10-01 07:19:47.060 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> 2.2a7 repair 11 errors, 0 fixed
> [root@ceph-n10 ~]#
> 
> [root@ceph-s1 ~]#  ceph pg repair 2.36b
> instructing pg 2.36bs0 on osd.254 to repair
> 
> [root@ceph-n29 ~]# grep "2.36b" /var/log/ceph/ceph-osd.254.log
> 2019-10-01 11:15:12.215 7fa01f589700  0 log_channel(cluster) log [DBG] :
> 2.36b repair starts
> 2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
> 2.36b shard 254(0) soid 2:d6cac754:::100070209f6.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
> 2.36b shard 10(2) soid 2:d6cac754:::100070209f6.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
> 2.36b shard 32(4) soid 2:d6cac754:::100070209f6.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
> 2.36b shard 53(9) soid 2:d6cac754:::100070209f6.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
> 2.36b shard 114(7) soid 2:d6cac754:::100070209f6.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
> 2.36b shard 169(8) soid 2:d6cac754:::100070209f6.:head :
> candidate size 4096 info size 0 mismatch
> 2019-10-01 

Re: [ceph-users] Nautilus minor versions archive

[Volodymyr Litovka]

It's quite pity. We tested solution Ceph 14.2.3 + RDMA in lab, got it
working, but at the moment of moving it to the production there was
14.2.4 and we got few hours outage trying to launch.

On 02.10.2019 00:28, Paul Emmerich wrote:

On Tue, Oct 1, 2019 at 11:21 PM Volodymyr Litovka  wrote:

Dear colleagues,

is archive of previos ceph minor versions  is available?

no, because reprepro doesn't support that

(yeah, that's clearly a solvable problem, but the current workflow
just uses reprepro and one repository)

Paul



To be more specific - I'm looking for 14.2.1 or 14.2.2 for Ubuntu, but
@download.ceph.com only 14.2.4 is available.

Thank you.

--
Volodymyr Litovka
"Vision without Execution is Hallucination." -- Thomas Edison

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


--
Volodymyr Litovka
  "Vision without Execution is Hallucination." -- Thomas Edison

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Nautilus minor versions archive

[Volodymyr Litovka]

On 02.10.2019 00:56, Paul Emmerich wrote:

There's virtually no difference between 14.2.3 and 14.2.4, it's only a
bug fix for running ceph-volume without a terminal on stderr on some
environments.


Nevetheless. There is one more minor change - kernel 5.0.xx -> 5.0.yy
(which probably can imply minor changes in Mellanox drivers). And yeah,
there are different CPUs :) and these changes resulted in continuous
kernel panic on different nodes.


Unrelated: Do you really want to run the RDMA messenger in production?


Not from today :-) It seems we'll stop any experiments with RDMA.

Thank you.

--
Volodymyr Litovka
  "Vision without Execution is Hallucination." -- Thomas Edison

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Commit and Apply latency on nautilus

[Sasha Litvak]

All,

Thank you for your suggestions.  During the last night test, I had at least
one drive on one node doing a power-on reset by the controller.   It caused
a couple of OSDs asserting / timing out on that node.  I am testing and
updating the usual suspects on this node and after that on a whole cluster,
i.e. kernel, controller firmware, SSD firmware. All of these have updates
available.  Dell mentioned a possible crush on bionic during high
throughput but none of it is clear and simple.   I would like to eliminate
firmware/drivers, especially if there is a bug causing a crash under the
load.  I will then proceed with Mokhtar's and Robert's suggestions.

If anyone has any more suggestions please share on this thread as it may
help someone else later on.

Best,

On Tue, Oct 1, 2019 at 2:56 PM Maged Mokhtar  wrote:

> Some suggestions:
>
> monitor raw resources such as cpu %util raw disk %util/busy, raw disk iops.
>
> instead of running a mix of workloads at this stage, narrow it down first,
> for example using rbd rand writes and 4k block sizes, then change 1 param
> at a time for example change the block size. See how your cluster performs
> and what resources loads you get step by step. Latency from 4M will not be
> the same as 4k.
>
> i would also run fio tests on the raw Nytro 1551 devices including sync
> writes.
>
> I would not recommend you increase readahead for random io.
>
> I do not recommend making RAID0
>
> /Maged
>
>
> On 01/10/2019 02:12, Sasha Litvak wrote:
>
> At this point, I ran out of ideas.  I changed nr_requests and readahead
> parameters to 128->1024 and 128->4096, tuned nodes to
> performance-throughput.  However, I still get high latency during benchmark
> testing.  I attempted to disable cache on ssd
>
> for i in {a..f}; do hdparm -W 0 -A 0 /dev/sd$i; done
>
> and I think it make things not better at all.  I have H740 and H730
> controllers with drives in HBA mode.
>
> Other them converting them one by one to RAID0 I am not sure what else I
> can try.
>
> Any suggestions?
>
>
> On Mon, Sep 30, 2019 at 2:45 PM Paul Emmerich 
> wrote:
>
>> BTW: commit and apply latency are the exact same thing since
>> BlueStore, so don't bother looking at both.
>>
>> In fact you should mostly be looking at the op_*_latency counters
>>
>>
>> Paul
>>
>> --
>> Paul Emmerich
>>
>> Looking for help with your Ceph cluster? Contact us at https://croit.io
>>
>> croit GmbH
>> Freseniusstr. 31h
>> 81247 München
>> www.croit.io
>> Tel: +49 89 1896585 90
>>
>> On Mon, Sep 30, 2019 at 8:46 PM Sasha Litvak
>>  wrote:
>> >
>> > In my case, I am using premade Prometheus sourced dashboards in grafana.
>> >
>> > For individual latency, the query looks like that
>> >
>> >  irate(ceph_osd_op_r_latency_sum{ceph_daemon=~"$osd"}[1m]) / on
>> (ceph_daemon) irate(ceph_osd_op_r_latency_count[1m])
>> > irate(ceph_osd_op_w_latency_sum{ceph_daemon=~"$osd"}[1m]) / on
>> (ceph_daemon) irate(ceph_osd_op_w_latency_count[1m])
>> >
>> > The other ones use
>> >
>> > ceph_osd_commit_latency_ms
>> > ceph_osd_apply_latency_ms
>> >
>> > and graph the distribution of it over time
>> >
>> > Also, average OSD op latency
>> >
>> > avg(rate(ceph_osd_op_r_latency_sum{cluster="$cluster"}[5m]) /
>> rate(ceph_osd_op_r_latency_count{cluster="$cluster"}[5m]) >= 0)
>> > avg(rate(ceph_osd_op_w_latency_sum{cluster="$cluster"}[5m]) /
>> rate(ceph_osd_op_w_latency_count{cluster="$cluster"}[5m]) >= 0)
>> >
>> > Average OSD apply + commit latency
>> > avg(ceph_osd_apply_latency_ms{cluster="$cluster"})
>> > avg(ceph_osd_commit_latency_ms{cluster="$cluster"})
>> >
>> >
>> > On Mon, Sep 30, 2019 at 11:13 AM Marc Roos 
>> wrote:
>> >>
>> >>
>> >> What parameters are you exactly using? I want to do a similar test on
>> >> luminous, before I upgrade to Nautilus. I have quite a lot (74+)
>> >>
>> >> type_instance=Osd.opBeforeDequeueOpLat
>> >> type_instance=Osd.opBeforeQueueOpLat
>> >> type_instance=Osd.opLatency
>> >> type_instance=Osd.opPrepareLatency
>> >> type_instance=Osd.opProcessLatency
>> >> type_instance=Osd.opRLatency
>> >> type_instance=Osd.opRPrepareLatency
>> >> type_instance=Osd.opRProcessLatency
>> >> type_instance=Osd.opRwLatency
>> >> type_instance=Osd.opRwPrepareLatency
>> >> type_instance=Osd.opRwProcessLatency
>> >> type_instance=Osd.opWLatency
>> >> type_instance=Osd.opWPrepareLatency
>> >> type_instance=Osd.opWProcessLatency
>> >> type_instance=Osd.subopLatency
>> >> type_instance=Osd.subopWLatency
>> >> ...
>> >> ...
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> -Original Message-
>> >> From: Alex Litvak [mailto:alexander.v.lit...@gmail.com]
>> >> Sent: zondag 29 september 2019 13:06
>> >> To: ceph-users@lists.ceph.com
>> >> Cc: ceph-de...@vger.kernel.org
>> >> Subject: [ceph-users] Commit and Apply latency on nautilus
>> >>
>> >> Hello everyone,
>> >>
>> >> I am running a number of parallel benchmark tests against the cluster
>> >> that should be ready to go to production.
>> >> I enabled prometheus to monitor various information and while 

Re: [ceph-users] NFS

[Brent Kennedy]

We might have to backup a step here so I can understand.  Are you saying stand 
up a new VM with just those packages installed, then configure the export file  
( the file location isn’t mentioned in the ceph docs ) and supposedly a client 
can connect to them?  ( only linux clients or any NFS client? )

I don’t use cephFS, so being that it will be an object storage backend, will 
that be ok with multiple hosts accessing files through the NFS one gateway or 
should I configure multiple gateways ( one for each share )?  

I was hoping to save large files( 20+ GB ), should I stand up cephFS instead 
for this?

I am used to using a NAS storage appliance server(or freeNAS ), so using ceph 
as a NAS backend is new to me ( thus I might be over thinking this )  :)

-Brent

-Original Message-
From: Daniel Gryniewicz  
Sent: Tuesday, October 1, 2019 8:20 AM
To: Marc Roos ; bkennedy ; 
ceph-users 
Subject: Re: [ceph-users] NFS

Ganesha can export CephFS or RGW.  It cannot export anything else (like iscsi 
or RBD).  Config for RGW looks like this:

EXPORT
{
 Export_ID=1;
 Path = "/";
 Pseudo = "/rgw";
 Access_Type = RW;
 Protocols = 4;
 Transports = TCP;
 FSAL {
 Name = RGW;
 User_Id = "testuser";
 Access_Key_Id ="";
 Secret_Access_Key = "";
 }
}

RGW {
 ceph_conf = "//ceph.conf";
 # for vstart cluster, name = "client.admin"
 name = "client.rgw.foohost";
 cluster = "ceph";
#   init_args = "-d --debug-rgw=16";
}


Daniel

On 9/30/19 3:01 PM, Marc Roos wrote:
>   
> Just install these
> 
> http://download.ceph.com/nfs-ganesha/
> nfs-ganesha-rgw-2.7.1-0.1.el7.x86_64
> nfs-ganesha-vfs-2.7.1-0.1.el7.x86_64
> libnfsidmap-0.25-19.el7.x86_64
> nfs-ganesha-mem-2.7.1-0.1.el7.x86_64
> nfs-ganesha-xfs-2.7.1-0.1.el7.x86_64
> nfs-ganesha-2.7.1-0.1.el7.x86_64
> nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64
> 
> 
> And export your cephfs like this:
> EXPORT {
>  Export_Id = 10;
>  Path = /nfs/cblr-repos;
>  Pseudo = /cblr-repos;
>  FSAL { Name = CEPH; User_Id = "cephfs.nfs.cblr"; 
> Secret_Access_Key = "xxx"; }
>  Disable_ACL = FALSE;
>  CLIENT { Clients = 192.168.10.2; access_type = "RW"; }
>  CLIENT { Clients = 192.168.10.253; } }
> 
> 
> -Original Message-
> From: Brent Kennedy [mailto:bkenn...@cfl.rr.com]
> Sent: maandag 30 september 2019 20:56
> To: 'ceph-users'
> Subject: [ceph-users] NFS
> 
> Wondering if there are any documents for standing up NFS with an 
> existing ceph cluster.  We don’t use ceph-ansible or any other tools 
> besides ceph-deploy.  The iscsi directions were pretty good once I got 
> past the dependencies.
> 
>   
> 
> I saw the one based on Rook, but it doesn’t seem to apply to our setup 
> of ceph vms with physical hosts doing OSDs.  The official ceph 
> documents talk about using ganesha but doesn’t seem to dive into the 
> details of what the process is for getting it online.  We don’t use 
> cephfs, so that’s not setup either.  The basic docs seem to note this is 
> required.
>   Seems my google-fu is failing me when I try to find a more 
> definitive guide.
> 
>   
> 
> The servers are all centos 7 with the latest updates.
> 
>   
> 
> Any guidance would be greatly appreciated!
> 
>   
> 
> Regards,
> 
> -Brent
> 
>   
> 
> Existing Clusters:
> 
> Test: Nautilus 14.2.2 with 3 osd servers, 1 mon/man, 1 gateway, 2 
> iscsi gateways ( all virtual on nvme )
> 
> US Production(HDD): Nautilus 14.2.2 with 13 osd servers, 3 mons, 4 
> gateways, 2 iscsi gateways
> 
> UK Production(HDD): Nautilus 14.2.2 with 25 osd servers, 3 mons/man, 3 
> gateways behind
> 
> US Production(SSD): Nautilus 14.2.2 with 6 osd servers, 3 mons/man, 3 
> gateways, 2 iscsi gateways
> 
>   
> 
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Issues with data distribution on Nautilus / weird filling behavior

[Philippe D'Anjou]

Hi,this is a fresh Nautilus cluster, but there is a second old one that was 
upgraded from Luminous to Nautilus, both experience the same symptoms.
First of all the data distribution on the OSDs is very bad. Now that could be 
due to low PGs although I get no recommendation to raise the PG number so far.

 14   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB  68 KiB  15 GiB 2.9 TiB 
67.83 1.07  32 up 
 19   hdd  9.09470  1.0 9.1 TiB 6.7 TiB 6.7 TiB  56 KiB  16 GiB 2.4 TiB 
73.56 1.16  35 up 
 22   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  88 KiB  15 GiB 2.8 TiB 
69.39 1.09  32 up 
 25   hdd  9.09470  1.0 9.1 TiB 6.0 TiB 6.0 TiB  88 KiB  16 GiB 3.1 TiB 
66.01 1.04  31 up 
 30   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  84 KiB  16 GiB 2.8 TiB 
69.36 1.09  33 up 
 33   hdd  9.09470  1.0 9.1 TiB 6.0 TiB 6.0 TiB  60 KiB  15 GiB 3.1 TiB 
65.84 1.03  31 up 
 34   hdd  9.09470  1.0 9.1 TiB 5.8 TiB 5.8 TiB 124 KiB  14 GiB 3.3 TiB 
63.84 1.00  29 up 
 35   hdd  9.09470  1.0 9.1 TiB 6.0 TiB 6.0 TiB  32 KiB  15 GiB 3.1 TiB 
65.82 1.03  31 up 
 12   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB  44 KiB  15 GiB 3.0 TiB 
67.32 1.06  31 up 
 16   hdd  9.09470  1.0 9.1 TiB 6.7 TiB 6.7 TiB  72 KiB  17 GiB 2.4 TiB 
73.52 1.16  35 up 
 17   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB  80 KiB  15 GiB 3.0 TiB 
67.24 1.06  32 up 
 20   hdd  9.09470  1.0 9.1 TiB 6.8 TiB 6.8 TiB  64 KiB  17 GiB 2.3 TiB 
74.45 1.17  35 up 
 23   hdd  9.09470  1.0 9.1 TiB 6.5 TiB 6.5 TiB  80 KiB  16 GiB 2.6 TiB 
71.46 1.12  34 up 
 26   hdd  9.09470  1.0 9.1 TiB 6.4 TiB 6.4 TiB  68 KiB  16 GiB 2.7 TiB 
70.45 1.11  33 up 
 28   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.0 TiB  64 KiB  15 GiB 3.0 TiB 
66.52 1.05  31 up 
 31   hdd  9.09470  1.0 9.1 TiB 6.6 TiB 6.6 TiB  52 KiB  16 GiB 2.5 TiB 
72.21 1.13  34 up 
 13   hdd  9.09470  1.0 9.1 TiB 6.7 TiB 6.7 TiB  68 KiB  16 GiB 2.4 TiB 
73.56 1.16  35 up 
 15   hdd  9.09470  1.0 9.1 TiB 6.7 TiB 6.7 TiB  44 KiB  17 GiB 2.4 TiB 
73.58 1.16  35 up 
 18   hdd  9.09470  1.0 9.1 TiB 6.5 TiB 6.5 TiB  56 KiB  16 GiB 2.6 TiB 
71.46 1.12  34 up 
 21   hdd  8.0  1.0 9.1 TiB 5.9 TiB 5.9 TiB  84 KiB  15 GiB 3.2 TiB 
65.14 1.02  31 up 
 24   hdd  9.09470  1.0 9.1 TiB 6.6 TiB 6.6 TiB  76 KiB  16 GiB 2.5 TiB 
72.21 1.13  34 up 
 27   hdd  9.09470  1.0 9.1 TiB 6.5 TiB 6.5 TiB  64 KiB  16 GiB 2.6 TiB 
71.42 1.12  34 up 
 29   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB  80 KiB  15 GiB 2.9 TiB 
68.16 1.07  32 up 
 32   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB 132 KiB  15 GiB 3.0 TiB 
66.91 1.05  31 up 
 37   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  16 KiB  17 GiB 2.8 TiB 
69.38 1.09  33 up 
 39   hdd  9.09470  1.0 9.1 TiB 6.4 TiB 6.4 TiB  92 KiB  16 GiB 2.7 TiB 
70.44 1.11  33 up 
 41   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB 108 KiB  15 GiB 2.9 TiB 
67.95 1.07  32 up 
 43   hdd  9.09470  1.0 9.1 TiB 5.9 TiB 5.9 TiB  24 KiB  15 GiB 3.2 TiB 
65.20 1.02  31 up 
 45   hdd  9.09470  1.0 9.1 TiB 5.9 TiB 5.9 TiB  72 KiB  15 GiB 3.2 TiB 
65.35 1.03  31 up 
 46   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB  60 KiB  15 GiB 2.9 TiB 
68.08 1.07  32 up 
 48   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB  64 KiB  15 GiB 2.9 TiB 
67.96 1.07  32 up 
 50   hdd  9.09470  1.0 9.1 TiB 5.7 TiB 5.7 TiB  48 KiB  15 GiB 3.4 TiB 
63.06 0.99  30 up 
 36   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB 116 KiB  15 GiB 3.0 TiB 
67.45 1.06  32 up 
 38   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  80 KiB  16 GiB 2.8 TiB 
69.36 1.09  33 up 
 40   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB  84 KiB  15 GiB 3.0 TiB 
67.31 1.06  32 up 
 42   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.0 TiB 104 KiB  15 GiB 3.0 TiB 
66.59 1.05  31 up 
 44   hdd  9.09470  1.0 9.1 TiB 5.9 TiB 5.9 TiB  68 KiB  15 GiB 3.2 TiB 
65.12 1.02  31 up 
 47   hdd  9.09470  1.0 9.1 TiB 5.9 TiB 5.9 TiB  88 KiB  15 GiB 3.2 TiB 
65.09 1.02  31 up 
 49   hdd  9.09470  1.0 9.1 TiB 6.1 TiB 6.1 TiB 112 KiB  15 GiB 3.0 TiB 
67.19 1.06  32 up 
 51   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.1 TiB  40 KiB  15 GiB 2.9 TiB 
67.75 1.06  32 up 
 52   hdd  9.09470  1.0 9.1 TiB 5.8 TiB 5.8 TiB  80 KiB  15 GiB 3.3 TiB 
63.91 1.00  30 up 
 53   hdd  7.0  1.0 9.1 TiB 7.3 TiB 7.2 TiB  88 KiB  18 GiB 1.8 TiB 
79.84 1.25  38 up 
 54   hdd  9.09470  1.0 9.1 TiB 5.8 TiB 5.8 TiB  96 KiB  14 GiB 3.3 TiB 
63.67 1.00  30 up 
 55   hdd  9.09470  1.0 9.1 TiB 5.3 TiB 5.3 TiB  48 KiB  13 GiB 3.8 TiB 
58.20 0.91  27 up 
 56   hdd  9.09470  1.0 9.1 TiB 6.3 TiB 6.3 TiB  64 KiB  16 GiB 2.8 TiB 
69.35 1.09  33 up 
 57   hdd  6.0  1.0 9.1 TiB 4.2 TiB 4.2 TiB  36 KiB  11 GiB 4.9 TiB 
46.27 0.73  22 up 
 58   hdd  9.09470  1.0 9.1 TiB 6.2 TiB 6.2 TiB 104 KiB  15 GiB 2.9 TiB 
68.20 1.07  32 

Re: [ceph-users] ceph pg repair fails...?

[Brad Hubbard]

On Wed, Oct 2, 2019 at 1:15 AM Mattia Belluco  wrote:
>
> Hi Jake,
>
> I am curious to see if your problem is similar to ours (despite the fact
> we are still on Luminous).
>
> Could you post the output of:
>
> rados list-inconsistent-obj 
>
> and
>
> rados list-inconsistent-snapset 

Make sure you scrub the pg before running these commands.
Take a look at the information in http://tracker.ceph.com/issues/24994
for hints on how to proceed.
'
>
> Thanks,
>
> Mattia
>
> On 10/1/19 1:08 PM, Jake Grimmett wrote:
> > Dear All,
> >
> > I've just found two inconsistent pg that fail to repair.
> >
> > This might be the same bug as shown here:
> >
> > 
> >
> > Cluster is running Nautilus 14.2.2
> > OS is Scientific Linux 7.6
> > DB/WAL on NVMe, Data on 12TB HDD
> >
> > Logs below cab also be seen here: 
> >
> > [root@ceph-s1 ~]# ceph health detail
> > HEALTH_ERR 22 scrub errors; Possible data damage: 2 pgs inconsistent
> > OSD_SCRUB_ERRORS 22 scrub errors
> > PG_DAMAGED Possible data damage: 2 pgs inconsistent
> > pg 2.2a7 is active+clean+inconsistent+failed_repair, acting
> > [83,60,133,326,281,162,180,172,144,219]
> > pg 2.36b is active+clean+inconsistent+failed_repair, acting
> > [254,268,10,262,32,280,211,114,169,53]
> >
> > Issued "pg repair" commands, osd log shows:
> > [root@ceph-n10 ~]# grep "2.2a7" /var/log/ceph/ceph-osd.83.log
> > 2019-10-01 07:05:02.459 7f9adab4b700  0 log_channel(cluster) log [DBG] :
> > 2.2a7 repair starts
> > 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> > 2.2a7 shard 83(0) soid 2:e5472cab:::1000702081f.:head :
> > candidate size 4096 info size 0 mismatch
> > 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> > 2.2a7 shard 60(1) soid 2:e5472cab:::1000702081f.:head :
> > candidate size 4096 info size 0 mismatch
> > 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> > 2.2a7 shard 133(2) soid 2:e5472cab:::1000702081f.:head :
> > candidate size 4096 info size 0 mismatch
> > 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> > 2.2a7 shard 144(8) soid 2:e5472cab:::1000702081f.:head :
> > candidate size 4096 info size 0 mismatch
> > 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> > 2.2a7 shard 162(5) soid 2:e5472cab:::1000702081f.:head :
> > candidate size 4096 info size 0 mismatch
> > 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> > 2.2a7 shard 172(7) soid 2:e5472cab:::1000702081f.:head :
> > candidate size 4096 info size 0 mismatch
> > 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> > 2.2a7 shard 180(6) soid 2:e5472cab:::1000702081f.:head :
> > candidate size 4096 info size 0 mismatch
> > 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> > 2.2a7 shard 219(9) soid 2:e5472cab:::1000702081f.:head :
> > candidate size 4096 info size 0 mismatch
> > 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> > 2.2a7 shard 281(4) soid 2:e5472cab:::1000702081f.:head :
> > candidate size 4096 info size 0 mismatch
> > 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> > 2.2a7 shard 326(3) soid 2:e5472cab:::1000702081f.:head :
> > candidate size 4096 info size 0 mismatch
> > 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> > 2.2a7 soid 2:e5472cab:::1000702081f.:head : failed to pick
> > suitable object info
> > 2019-10-01 07:11:41.589 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> > repair 2.2a7s0 2:e5472cab:::1000702081f.:head : on disk size
> > (4096) does not match object info size (0) adjusted for ondisk to (0)
> > 2019-10-01 07:19:47.060 7f9adab4b700 -1 log_channel(cluster) log [ERR] :
> > 2.2a7 repair 11 errors, 0 fixed
> > [root@ceph-n10 ~]#
> >
> > [root@ceph-s1 ~]#  ceph pg repair 2.36b
> > instructing pg 2.36bs0 on osd.254 to repair
> >
> > [root@ceph-n29 ~]# grep "2.36b" /var/log/ceph/ceph-osd.254.log
> > 2019-10-01 11:15:12.215 7fa01f589700  0 log_channel(cluster) log [DBG] :
> > 2.36b repair starts
> > 2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
> > 2.36b shard 254(0) soid 2:d6cac754:::100070209f6.:head :
> > candidate size 4096 info size 0 mismatch
> > 2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
> > 2.36b shard 10(2) soid 2:d6cac754:::100070209f6.:head :
> > candidate size 4096 info size 0 mismatch
> > 2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
> > 2.36b shard 32(4) soid 2:d6cac754:::100070209f6.:head :
> > candidate size 4096 info size 0 mismatch
> > 2019-10-01 11:25:12.241 7fa01f589700 -1 log_channel(cluster) log [ERR] :
> > 2.36b shard 53(9) soid 2:d6cac754:::100070209f6.:head :
> > candidate size 4096 info 

Re: [ceph-users] Nautilus minor versions archive

[Paul Emmerich]

There's virtually no difference between 14.2.3 and 14.2.4, it's only a
bug fix for running ceph-volume without a terminal on stderr on some
environments.

Unrelated: Do you really want to run the RDMA messenger in production?
I wouldn't dare if the system is in any way critical.

Paul

-- 
Paul Emmerich

Looking for help with your Ceph cluster? Contact us at https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io
Tel: +49 89 1896585 90

On Tue, Oct 1, 2019 at 11:48 PM Volodymyr Litovka  wrote:
>
> It's quite pity. We tested solution Ceph 14.2.3 + RDMA in lab, got it
> working, but at the moment of moving it to the production there was
> 14.2.4 and we got few hours outage trying to launch.
>
> On 02.10.2019 00:28, Paul Emmerich wrote:
> > On Tue, Oct 1, 2019 at 11:21 PM Volodymyr Litovka  wrote:
> >> Dear colleagues,
> >>
> >> is archive of previos ceph minor versions  is available?
> > no, because reprepro doesn't support that
> >
> > (yeah, that's clearly a solvable problem, but the current workflow
> > just uses reprepro and one repository)
> >
> > Paul
> >
> >
> >> To be more specific - I'm looking for 14.2.1 or 14.2.2 for Ubuntu, but
> >> @download.ceph.com only 14.2.4 is available.
> >>
> >> Thank you.
> >>
> >> --
> >> Volodymyr Litovka
> >> "Vision without Execution is Hallucination." -- Thomas Edison
> >>
> >> ___
> >> ceph-users mailing list
> >> ceph-users@lists.ceph.com
> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
> --
> Volodymyr Litovka
>"Vision without Execution is Hallucination." -- Thomas Edison
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] OSD crashed during the fio test

[Brad Hubbard]

If it is only this one osd I'd be inclined to be taking a hard look at
the underlying hardware and how it behaves/performs compared to the hw
backing identical osds. The less likely possibility is that you have
some sort of "hot spot" causing resource contention for that osd. To
investigate that further you could look at whether the pattern of cpu
and ram usage of that daemon varies significantly compared to the
other osd daemons in the cluster. You could also compare perf dumps
between daemons.

On Wed, Oct 2, 2019 at 1:46 PM Sasha Litvak
 wrote:
>
> I updated firmware and kernel, running torture tests.  So far no assert, but 
> I still noticed this on the same osd as yesterday
>
> Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721 
> 7f8d03150700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 
> 0x7f8cd05d7700' had timed out after 60
> Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721 
> 7f8d03150700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 
> 0x7f8cd0dd8700' had timed out after 60
> Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721 
> 7f8d03150700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 
> 0x7f8cd2ddc700' had timed out after 60
> Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721 
> 7f8d03150700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 
> 0x7f8cd35dd700' had timed out after 60
> Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721 
> 7f8d03150700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 
> 0x7f8cd3dde700' had timed out after 60
>
> The spike of latency on this OSD is 6 seconds at that time.  Any ideas?
>
> On Tue, Oct 1, 2019 at 8:03 AM Sasha Litvak  
> wrote:
>>
>> It was hardware indeed.  Dell server reported a disk being reset with power 
>> on.  Checking the usual suspects i.e. controller firmware, controller event 
>> log (if I can get one), drive firmware.
>> I will report more when I get a better idea
>>
>> Thank you!
>>
>> On Tue, Oct 1, 2019 at 2:33 AM Brad Hubbard  wrote:
>>>
>>> Removed ceph-de...@vger.kernel.org and added d...@ceph.io
>>>
>>> On Tue, Oct 1, 2019 at 4:26 PM Alex Litvak  
>>> wrote:
>>> >
>>> > Hellow everyone,
>>> >
>>> > Can you shed the line on the cause of the crash?  Could actually client 
>>> > request trigger it?
>>> >
>>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 22:52:58.867 
>>> > 7f093d71e700 -1 bdev(0x55b72c156000 /var/lib/ceph/osd/ceph-17/block) 
>>> > aio_submit retries 16
>>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 22:52:58.867 
>>> > 7f093d71e700 -1 bdev(0x55b72c156000 /var/lib/ceph/osd/ceph-17/block)  aio 
>>> > submit got (11) Resource temporarily unavailable
>>>
>>> The KernelDevice::aio_submit function has tried to submit Io 16 times
>>> (a hard coded limit) and received an error each time causing it to
>>> assert. Can you check the status of the underlying device(s)?
>>>
>>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:
>>> > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc:
>>> > In fun
>>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:
>>> > /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc:
>>> > 757: F
>>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  ceph version 14.2.2 
>>> > (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
>>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  1: 
>>> > (ceph::__ceph_assert_fail(char const*, char const*, int, char 
>>> > const*)+0x14a) [0x55b71f668cf4]
>>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  2: 
>>> > (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, 
>>> > char const*, ...)+0) [0x55b71f668ec2]
>>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  3: 
>>> > (KernelDevice::aio_submit(IOContext*)+0x701) [0x55b71fd61ca1]
>>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  4: 
>>> > (BlueStore::_txc_aio_submit(BlueStore::TransContext*)+0x42) 
>>> > [0x55b71fc29892]
>>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  5: 
>>> > (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x42b) 
>>> > [0x55b71fc496ab]
>>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  6: 
>>> > (BlueStore::queue_transactions(boost::intrusive_ptr&,
>>> >  std::vector>> > std::allocator >&, 
>>> > boost::intrusive_ptr, ThreadPool::T
>>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  7: (non-virtual thunk 
>>> > to PrimaryLogPG::queue_transactions(std::vector>> > std::allocator >&,
>>> > boost::intrusive_ptr)+0x54) [0x55b71f9b1b84]
>>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  8: 
>>> > 

Re: [ceph-users] OSD crashed during the fio test

[Sasha Litvak]

I updated firmware and kernel, running torture tests.  So far no assert,
but I still noticed this on the same osd as yesterday

Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721
7f8d03150700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread
0x7f8cd05d7700' had timed out after 60
Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721
7f8d03150700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread
0x7f8cd0dd8700' had timed out after 60
Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721
7f8d03150700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread
0x7f8cd2ddc700' had timed out after 60
Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721
7f8d03150700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread
0x7f8cd35dd700' had timed out after 60
Oct 01 19:35:13 storage2n2-la ceph-osd-34[11188]: 2019-10-01 19:35:13.721
7f8d03150700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread
0x7f8cd3dde700' had timed out after 60

The spike of latency on this OSD is 6 seconds at that time.  Any ideas?

On Tue, Oct 1, 2019 at 8:03 AM Sasha Litvak 
wrote:

> It was hardware indeed.  Dell server reported a disk being reset with
> power on.  Checking the usual suspects i.e. controller firmware, controller
> event log (if I can get one), drive firmware.
> I will report more when I get a better idea
>
> Thank you!
>
> On Tue, Oct 1, 2019 at 2:33 AM Brad Hubbard  wrote:
>
>> Removed ceph-de...@vger.kernel.org and added d...@ceph.io
>>
>> On Tue, Oct 1, 2019 at 4:26 PM Alex Litvak 
>> wrote:
>> >
>> > Hellow everyone,
>> >
>> > Can you shed the line on the cause of the crash?  Could actually client
>> request trigger it?
>> >
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30
>> 22:52:58.867 7f093d71e700 -1 bdev(0x55b72c156000
>> /var/lib/ceph/osd/ceph-17/block) aio_submit retries 16
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30
>> 22:52:58.867 7f093d71e700 -1 bdev(0x55b72c156000
>> /var/lib/ceph/osd/ceph-17/block)  aio submit got (11) Resource temporarily
>> unavailable
>>
>> The KernelDevice::aio_submit function has tried to submit Io 16 times
>> (a hard coded limit) and received an error each time causing it to
>> assert. Can you check the status of the underlying device(s)?
>>
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:
>> >
>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc:
>> > In fun
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:
>> >
>> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc:
>> > 757: F
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  ceph version 14.2.2
>> (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  1:
>> (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x14a) [0x55b71f668cf4]
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  2:
>> (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char
>> const*, ...)+0) [0x55b71f668ec2]
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  3:
>> (KernelDevice::aio_submit(IOContext*)+0x701) [0x55b71fd61ca1]
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  4:
>> (BlueStore::_txc_aio_submit(BlueStore::TransContext*)+0x42) [0x55b71fc29892]
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  5:
>> (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x42b)
>> [0x55b71fc496ab]
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  6:
>> (BlueStore::queue_transactions(boost::intrusive_ptr&,
>> std::vector> > std::allocator >&,
>> boost::intrusive_ptr, ThreadPool::T
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  7: (non-virtual
>> thunk to
>> PrimaryLogPG::queue_transactions(std::vector> std::allocator >&,
>> > boost::intrusive_ptr)+0x54) [0x55b71f9b1b84]
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  8:
>> (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t
>> const&, eversion_t const&, std::unique_ptr> > std::default_delete >&&, eversion_t const&, eversion_t
>> const&, s
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  9:
>> (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*,
>> PrimaryLogPG::OpContext*)+0xf12) [0x55b71f90e322]
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  10:
>> (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfae) [0x55b71f969b7e]
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  11:
>> (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x3965)
>> [0x55b71f96de15]
>> > Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  12:
>> (PrimaryLogPG::do_request(boost::intrusive_ptr&,
>> 

[ceph-users] hanging/stopped recovery/rebalance in Nautilus

[Philippe D'Anjou]

Hi,I often observed now that the recovery/rebalance in Nautilus starts quite 
fast but gets extremely slow (2-3 objects/s) even if there are like 20 OSDs 
involved. Right now I am moving (reweighted to 0) 16x8TB disks, it's running 
since 4 days and since 12h it's kind of stuck now at
  cluster:
    id: 2f525d60-aada-4da6-830f-7ba7b46c546b
    health: HEALTH_WARN
    Degraded data redundancy: 1070/899796274 objects degraded (0.000%), 
1 pg degraded, 1 pg undersized
    1216 pgs not deep-scrubbed in time
    1216 pgs not scrubbed in time
  
  services:
    mon: 1 daemons, quorum km-fsn-1-dc4-m1-797678 (age 8w)
    mgr: km-fsn-1-dc4-m1-797678(active, since 6w)
    mds: xfd:1 {0=km-fsn-1-dc4-m1-797678=up:active}
    osd: 151 osds: 151 up (since 3d), 151 in (since 7d); 24 remapped pgs
    rgw: 1 daemon active (km-fsn-1-dc4-m1-797680)
 
  data:
    pools:   13 pools, 10433 pgs
    objects: 447.45M objects, 282 TiB
    usage:   602 TiB used, 675 TiB / 1.2 PiB avail
    pgs: 1070/899796274 objects degraded (0.000%)
 261226/899796274 objects misplaced (0.029%)
 10388 active+clean
 24    active+clean+remapped
 19    active+clean+scrubbing+deep
 1 active+undersized+degraded
 1 active+clean+scrubbing
 
  io:
    client:   10 MiB/s rd, 18 MiB/s wr, 141 op/s rd, 292 op/s wr


osd-max-backfill is at 16 for all OSDs.
Anyone got an idea why rebalance completely stopped?
Thanks
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] MDS / CephFS behaviour with unusual directory layout

[Stefan Kooman]

Quoting Stefan Kooman (ste...@bit.nl):
> Hi List,
> 
> We are planning to move a filesystem workload (currently nfs) to CephFS.
> It's around 29 TB. The unusual thing here is the amount of directories
> in use to host the files. In order to combat a "too many files in one
> directory" scenario a "let's make use of recursive directories" approach.
> Not ideal either. This workload is supposed to be moved to (Ceph) S3
> sometime in the future, but until then, it has to go to a shared
> filesystem ...
> 
> So what is unusual about this? The directory layout looks like this
> 
> /data/files/00/00/[0-8][0-9]/[0-9]/ from this point on there will be 7
> directories created to store 1 file.
> 
> Total amount of directories in a file path is 14. There are around 150 M
> files in 400 M directories.
> 
> The working set won't be big. Most files will just sit around and will
> not be touched. The active amount of files wil be a few thousand.
> 
> We are wondering if this kind of directory structure is suitable for
> CephFS. Might the MDS get difficulties with keeping up that many inodes
> / dentries or doesn't it care at all?
> 
> The amount of metadata overhead might be horrible, but we will test that
> out.

This awkward dataset is "live" ... and the MDS has been happily
crunching away so far. Peaking at 42.5 M caps. Multiple parallel rsyncs
(20+) to fill cephfs was no issue whatsover.

Thanks Nathan Fish and Burkhard Linke for sharing helpful MDS insight!

Gr. Stefan

-- 
| BIT BV  https://www.bit.nl/Kamer van Koophandel 09090351
| GPG: 0xD14839C6   +31 318 648 688 / i...@bit.nl
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] ceph-osd@n crash dumps

[Brad Hubbard]

On Tue, Oct 1, 2019 at 10:43 PM Del Monaco, Andrea <
andrea.delmon...@atos.net> wrote:

> Hi list,
>
> After the nodes ran OOM and after reboot, we are not able to restart the
> ceph-osd@x services anymore. (Details about the setup at the end).
>
> I am trying to do this manually, so we can see the error but all i see is
> several crash dumps - this is just one of the OSDs which is not starting.
> Any idea how to get past this??
> [root@ceph001 ~]# /usr/bin/ceph-osd --debug_osd 10 -f --cluster ceph --id
> 83 --setuser ceph --setgroup ceph  > /tmp/dump 2>&1
> starting osd.83 at - osd_data /var/lib/ceph/osd/ceph-83
> /var/lib/ceph/osd/ceph-83/journal
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h:
> In function 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)'
> thread 2aaf5540 time 2019-10-01 14:19:49.494368
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h:
> 34: FAILED assert(stripe_width % stripe_size == 0)
>  ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic
> (stable)
>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x14b) [0x2af3d36b]
>  2: (()+0x26e4f7) [0x2af3d4f7]
>  3: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&,
> boost::intrusive_ptr&, ObjectStore*,
> CephContext*, std::shared_ptr, unsigned
> long)+0x46d) [0x55c0bd3d]
>  4: (PGBackend::build_pg_backend(pg_pool_t const&, std::map std::string, std::less, std::allocator const, std::string> > > const&, PGBackend::Listener*, coll_t,
> boost::intrusive_ptr&, ObjectStore*,
> CephContext*)+0x30a) [0x55b0ba8a]
>  5: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr const>, PGPool const&, std::map std::less, std::allocator std::string> > > const&, spg_t)+0x140) [0x55abd100]
>  6: (OSD::_make_pg(std::shared_ptr, spg_t)+0x10cb)
> [0x55914ecb]
>  7: (OSD::load_pgs()+0x4a9) [0x55917e39]
>  8: (OSD::init()+0xc99) [0x559238e9]
>  9: (main()+0x23a3) [0x558017a3]
>  10: (__libc_start_main()+0xf5) [0x2aaab77de495]
>  11: (()+0x385900) [0x558d9900]
> 2019-10-01 14:19:49.500 2aaf5540 -1
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h:
> In function 'ECUtil::stripe_info_t::stripe_info_t(uint64_t, uint64_t)'
> thread 2aaf5540 time 2019-10-01 14:19:49.494368
> /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/13.2.6/rpm/el7/BUILD/ceph-13.2.6/src/osd/ECUtil.h:
> 34: FAILED assert(stripe_width % stripe_size == 0)
>

 https://tracker.ceph.com/issues/41336 may be relevant here.

Can you post details of the pool involved as well as the erasure code
profile in use for that pool?


>  ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic
> (stable)
>  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x14b) [0x2af3d36b]
>  2: (()+0x26e4f7) [0x2af3d4f7]
>  3: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&,
> boost::intrusive_ptr&, ObjectStore*,
> CephContext*, std::shared_ptr, unsigned
> long)+0x46d) [0x55c0bd3d]
>  4: (PGBackend::build_pg_backend(pg_pool_t const&, std::map std::string, std::less, std::allocator const, std::string> > > const&, PGBackend::Listener*, coll_t,
> boost::intrusive_ptr&, ObjectStore*,
> CephContext*)+0x30a) [0x55b0ba8a]
>  5: (PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr const>, PGPool const&, std::map std::less, std::allocator std::string> > > const&, spg_t)+0x140) [0x55abd100]
>  6: (OSD::_make_pg(std::shared_ptr, spg_t)+0x10cb)
> [0x55914ecb]
>  7: (OSD::load_pgs()+0x4a9) [0x55917e39]
>  8: (OSD::init()+0xc99) [0x559238e9]
>  9: (main()+0x23a3) [0x558017a3]
>  10: (__libc_start_main()+0xf5) [0x2aaab77de495]
>  11: (()+0x385900) [0x558d9900]
>
> *** Caught signal (Aborted) **
>  in thread 2aaf5540 thread_name:ceph-osd
>  ceph version 13.2.6 (7b695f835b03642f85998b2ae7b6dd093d9fbce4) mimic
> (stable)
>  1: (()+0xf5d0) [0x2aaab69765d0]
>  2: (gsignal()+0x37) [0x2aaab77f22c7]
>  3: (abort()+0x148) [0x2aaab77f39b8]
>  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char
> const*)+0x248) [0x2af3d468]
>  5: (()+0x26e4f7) [0x2af3d4f7]
>  6: (ECBackend::ECBackend(PGBackend::Listener*, coll_t const&,
> boost::intrusive_ptr&, ObjectStore*,
> CephContext*, std::shared_ptr, unsigned
> long)+0x46d) [0x55c0bd3d]
>  7: (PGBackend::build_pg_backend(pg_pool_t const&, std::map std::string, std::less, std::allocator const, std::string> > > const&, PGBackend::Listener*, coll_t,
> 

Re: [ceph-users] how to set osd_crush_initial_weight 0 without restart any service

[Paul Mezzanini]

You could also:
ceph osd set norebalance


--
Paul Mezzanini
Sr Systems Administrator / Engineer, Research Computing
Information & Technology Services
Finance & Administration
Rochester Institute of Technology
o:(585) 475-3245 | pfm...@rit.edu

CONFIDENTIALITY NOTE: The information transmitted, including attachments, is
intended only for the person(s) or entity to which it is addressed and may
contain confidential and/or privileged material. Any review, retransmission,
dissemination or other use of, or taking of any action in reliance upon this
information by persons or entities other than the intended recipient is
prohibited. If you received this in error, please contact the sender and
destroy any copies of this information.



From: ceph-users  on behalf of Satish Patel 

Sent: Tuesday, October 1, 2019 2:34 PM
To: ceph-users
Subject: [ceph-users] how to set osd_crush_initial_weight 0 without restart any 
service

Folks,

Method: 1

In my lab i am playing with ceph and trying to understand how to add
new OSD without starting rebalancing.

I want to add this option on fly so i don't need to restart any
services or anything.

$ ceph tell mon.* injectargs '--osd_crush_initial_weight 0'

$ ceph daemon /var/run/ceph/ceph-mon.*.asok config show | grep
osd_crush_initial_weight
"osd_crush_initial_weight": "0.00",

All looks good, now i am adding OSD with ceph-ansible and you know
what look like it don't honer that option and adding OSD with default
weight (In my case i have 1.9TB SSD so weight is 1.7)

Can someone confirm injectargs work with osd_crush_initial_weight ?


Method: 2

Now i have added that option in ceph-ansible playbook like following

ceph_conf_overrides:
  osd:
osd_crush_initial_weight: 0

and i run playbook and it did magic and added OSD with weight zero (0)
 but i have notice it restarted all OSD daemon on that node, i am
worried is it safe to restart osd daemon on ceph in production?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Commit and Apply latency on nautilus

[Maged Mokhtar]

Some suggestions:

monitor raw resources such as cpu %util raw disk %util/busy, raw disk iops.

instead of running a mix of workloads at this stage, narrow it down 
first, for example using rbd rand writes and 4k block sizes, then change 
1 param at a time for example change the block size. See how your 
cluster performs and what resources loads you get step by step. Latency 
from 4M will not be the same as 4k.


i would also run fio tests on the raw Nytro 1551 devices including sync 
writes.


I would not recommend you increase readahead for random io.

I do not recommend making RAID0

/Maged


On 01/10/2019 02:12, Sasha Litvak wrote:
At this point, I ran out of ideas.  I changed nr_requests and 
readahead parameters to 128->1024 and 128->4096, tuned nodes to 
performance-throughput.  However, I still get high latency during 
benchmark testing.  I attempted to disable cache on ssd


for i in {a..f}; do hdparm -W 0 -A 0 /dev/sd$i; done

and I think it make things not better at all.  I have H740 and H730 
controllers with drives in HBA mode.


Other them converting them one by one to RAID0 I am not sure what else 
I can try.


Any suggestions?


On Mon, Sep 30, 2019 at 2:45 PM Paul Emmerich > wrote:


BTW: commit and apply latency are the exact same thing since
BlueStore, so don't bother looking at both.

In fact you should mostly be looking at the op_*_latency counters


Paul

-- 
Paul Emmerich


Looking for help with your Ceph cluster? Contact us at
https://croit.io

croit GmbH
Freseniusstr. 31h
81247 München
www.croit.io 
Tel: +49 89 1896585 90

On Mon, Sep 30, 2019 at 8:46 PM Sasha Litvak
mailto:alexander.v.lit...@gmail.com>> wrote:
>
> In my case, I am using premade Prometheus sourced dashboards in
grafana.
>
> For individual latency, the query looks like that
>
> irate(ceph_osd_op_r_latency_sum{ceph_daemon=~"$osd"}[1m]) / on
(ceph_daemon) irate(ceph_osd_op_r_latency_count[1m])
> irate(ceph_osd_op_w_latency_sum{ceph_daemon=~"$osd"}[1m]) / on
(ceph_daemon) irate(ceph_osd_op_w_latency_count[1m])
>
> The other ones use
>
> ceph_osd_commit_latency_ms
> ceph_osd_apply_latency_ms
>
> and graph the distribution of it over time
>
> Also, average OSD op latency
>
> avg(rate(ceph_osd_op_r_latency_sum{cluster="$cluster"}[5m]) /
rate(ceph_osd_op_r_latency_count{cluster="$cluster"}[5m]) >= 0)
> avg(rate(ceph_osd_op_w_latency_sum{cluster="$cluster"}[5m]) /
rate(ceph_osd_op_w_latency_count{cluster="$cluster"}[5m]) >= 0)
>
> Average OSD apply + commit latency
> avg(ceph_osd_apply_latency_ms{cluster="$cluster"})
> avg(ceph_osd_commit_latency_ms{cluster="$cluster"})
>
>
> On Mon, Sep 30, 2019 at 11:13 AM Marc Roos
mailto:m.r...@f1-outsourcing.eu>> wrote:
>>
>>
>> What parameters are you exactly using? I want to do a similar
test on
>> luminous, before I upgrade to Nautilus. I have quite a lot (74+)
>>
>> type_instance=Osd.opBeforeDequeueOpLat
>> type_instance=Osd.opBeforeQueueOpLat
>> type_instance=Osd.opLatency
>> type_instance=Osd.opPrepareLatency
>> type_instance=Osd.opProcessLatency
>> type_instance=Osd.opRLatency
>> type_instance=Osd.opRPrepareLatency
>> type_instance=Osd.opRProcessLatency
>> type_instance=Osd.opRwLatency
>> type_instance=Osd.opRwPrepareLatency
>> type_instance=Osd.opRwProcessLatency
>> type_instance=Osd.opWLatency
>> type_instance=Osd.opWPrepareLatency
>> type_instance=Osd.opWProcessLatency
>> type_instance=Osd.subopLatency
>> type_instance=Osd.subopWLatency
>> ...
>> ...
>>
>>
>>
>>
>>
>> -Original Message-
>> From: Alex Litvak [mailto:alexander.v.lit...@gmail.com
]
>> Sent: zondag 29 september 2019 13:06
>> To: ceph-users@lists.ceph.com 
>> Cc: ceph-de...@vger.kernel.org 
>> Subject: [ceph-users] Commit and Apply latency on nautilus
>>
>> Hello everyone,
>>
>> I am running a number of parallel benchmark tests against the
cluster
>> that should be ready to go to production.
>> I enabled prometheus to monitor various information and while
cluster
>> stays healthy through the tests with no errors or slow requests,
>> I noticed an apply / commit latency jumping between 40 - 600 ms on
>> multiple SSDs.  At the same time op_read and op_write are on
average
>> below 0.25 ms in the worth case scenario.
>>
>> I am running nautilus 14.2.2, all bluestore, no separate NVME
devices
>> for WAL/DB, 6 SSDs per node(Dell PowerEdge R440) with all
drives Seagate
>> Nytro 1551, osd spread across 6 nodes, running in
>> containers.  Each node has 

[ceph-users] Nautilus minor versions archive

[Volodymyr Litovka]

Dear colleagues,

is archive of previos ceph minor versions  is available?

To be more specific - I'm looking for 14.2.1 or 14.2.2 for Ubuntu, but
@download.ceph.com only 14.2.4 is available.

Thank you.

--
Volodymyr Litovka
  "Vision without Execution is Hallucination." -- Thomas Edison

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] how to set osd_crush_initial_weight 0 without restart any service

[Satish Patel]

Folks,

Method: 1

In my lab i am playing with ceph and trying to understand how to add
new OSD without starting rebalancing.

I want to add this option on fly so i don't need to restart any
services or anything.

$ ceph tell mon.* injectargs '--osd_crush_initial_weight 0'

$ ceph daemon /var/run/ceph/ceph-mon.*.asok config show | grep
osd_crush_initial_weight
"osd_crush_initial_weight": "0.00",

All looks good, now i am adding OSD with ceph-ansible and you know
what look like it don't honer that option and adding OSD with default
weight (In my case i have 1.9TB SSD so weight is 1.7)

Can someone confirm injectargs work with osd_crush_initial_weight ?


Method: 2

Now i have added that option in ceph-ansible playbook like following

ceph_conf_overrides:
  osd:
osd_crush_initial_weight: 0

and i run playbook and it did magic and added OSD with weight zero (0)
 but i have notice it restarted all OSD daemon on that node, i am
worried is it safe to restart osd daemon on ceph in production?
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to set osd_crush_initial_weight 0 without restart any service

[Satish Patel]

You are saying set "ceph osd set norebalance" before running
ceph-ansible playbook to add OSD

once osd visible in "ceph osd tree"  then i should do reweight to 0
and then do "ceph osd unset norebalance"

On Tue, Oct 1, 2019 at 2:41 PM Paul Mezzanini  wrote:
>
> You could also:
> ceph osd set norebalance
>
>
> --
> Paul Mezzanini
> Sr Systems Administrator / Engineer, Research Computing
> Information & Technology Services
> Finance & Administration
> Rochester Institute of Technology
> o:(585) 475-3245 | pfm...@rit.edu
>
> CONFIDENTIALITY NOTE: The information transmitted, including attachments, is
> intended only for the person(s) or entity to which it is addressed and may
> contain confidential and/or privileged material. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance upon this
> information by persons or entities other than the intended recipient is
> prohibited. If you received this in error, please contact the sender and
> destroy any copies of this information.
> 
>
> 
> From: ceph-users  on behalf of Satish 
> Patel 
> Sent: Tuesday, October 1, 2019 2:34 PM
> To: ceph-users
> Subject: [ceph-users] how to set osd_crush_initial_weight 0 without restart 
> any service
>
> Folks,
>
> Method: 1
>
> In my lab i am playing with ceph and trying to understand how to add
> new OSD without starting rebalancing.
>
> I want to add this option on fly so i don't need to restart any
> services or anything.
>
> $ ceph tell mon.* injectargs '--osd_crush_initial_weight 0'
>
> $ ceph daemon /var/run/ceph/ceph-mon.*.asok config show | grep
> osd_crush_initial_weight
> "osd_crush_initial_weight": "0.00",
>
> All looks good, now i am adding OSD with ceph-ansible and you know
> what look like it don't honer that option and adding OSD with default
> weight (In my case i have 1.9TB SSD so weight is 1.7)
>
> Can someone confirm injectargs work with osd_crush_initial_weight ?
>
>
> Method: 2
>
> Now i have added that option in ceph-ansible playbook like following
>
> ceph_conf_overrides:
>   osd:
> osd_crush_initial_weight: 0
>
> and i run playbook and it did magic and added OSD with weight zero (0)
>  but i have notice it restarted all OSD daemon on that node, i am
> worried is it safe to restart osd daemon on ceph in production?
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Panic in kernel CephFS client after kernel update

[Kenneth Van Alstyne]

All:
I’m not sure this should go to LKML or here, but I’ll start here.  After 
upgrading from Linux kernel 4.19.60 to 4.19.75 (or 76), I started running into 
kernel panics in the “ceph” module.  Based on the call trace, I believe I was 
able to narrow it down to the following commit in the Linux kernel 4.19 source 
tree:

commit 81281039a673d30f9d04d38659030a28051a
Author: Yan, Zheng mailto:z...@redhat.com>>
Date:   Sun Jun 2 09:45:38 2019 +0800

ceph: use ceph_evict_inode to cleanup inode's resource

[ Upstream commit 87bc5b895d94a0f40fe170d4cf5771c8e8f85d15 ]

remove_session_caps() relies on __wait_on_freeing_inode(), to wait for
freeing inode to remove its caps. But VFS wakes freeing inode waiters
before calling destroy_inode().

Cc: sta...@vger.kernel.org
Link: https://tracker.ceph.com/issues/40102
Signed-off-by: "Yan, Zheng" mailto:z...@redhat.com>>
Reviewed-by: Jeff Layton mailto:jlay...@redhat.com>>
Signed-off-by: Ilya Dryomov mailto:idryo...@gmail.com>>
Signed-off-by: Sasha Levin mailto:sas...@kernel.org>>


Backing this patch out and recompiling my kernel has since resolved my issues 
(as far as I can tell thus far).  The issue was fairly easy to create by simply 
creating and deleting files.  I tested using ‘dd’ and was pretty consistently 
able to reproduce the issue. Since the issue occurred in a VM, I do have a 
screenshot of the crashed machine and to avoid attaching an image, I’ll link to 
where they are:  http://kvanals.kvanals.org/.ceph_kernel_panic_images/

Am I way off base or has anyone else run into this issue?

Thanks,

--
Kenneth Van Alstyne
Systems Architect
M: 228.547.8045
15052 Conference Center Dr, Chantilly, VA 20151
perspecta

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] how to set osd_crush_initial_weight 0 without restart any service

[Satish Patel]

Paul,

I have tried your idea but it didn't work, i did set nobalance but it
still did rebalancing and fill lots of data on my new OSD.

I believe your option doesn't work with ceph-ansible playbook

On Tue, Oct 1, 2019 at 2:45 PM Satish Patel  wrote:
>
> You are saying set "ceph osd set norebalance" before running
> ceph-ansible playbook to add OSD
>
> once osd visible in "ceph osd tree"  then i should do reweight to 0
> and then do "ceph osd unset norebalance"
>
> On Tue, Oct 1, 2019 at 2:41 PM Paul Mezzanini  wrote:
> >
> > You could also:
> > ceph osd set norebalance
> >
> >
> > --
> > Paul Mezzanini
> > Sr Systems Administrator / Engineer, Research Computing
> > Information & Technology Services
> > Finance & Administration
> > Rochester Institute of Technology
> > o:(585) 475-3245 | pfm...@rit.edu
> >
> > CONFIDENTIALITY NOTE: The information transmitted, including attachments, is
> > intended only for the person(s) or entity to which it is addressed and may
> > contain confidential and/or privileged material. Any review, retransmission,
> > dissemination or other use of, or taking of any action in reliance upon this
> > information by persons or entities other than the intended recipient is
> > prohibited. If you received this in error, please contact the sender and
> > destroy any copies of this information.
> > 
> >
> > 
> > From: ceph-users  on behalf of Satish 
> > Patel 
> > Sent: Tuesday, October 1, 2019 2:34 PM
> > To: ceph-users
> > Subject: [ceph-users] how to set osd_crush_initial_weight 0 without restart 
> > any service
> >
> > Folks,
> >
> > Method: 1
> >
> > In my lab i am playing with ceph and trying to understand how to add
> > new OSD without starting rebalancing.
> >
> > I want to add this option on fly so i don't need to restart any
> > services or anything.
> >
> > $ ceph tell mon.* injectargs '--osd_crush_initial_weight 0'
> >
> > $ ceph daemon /var/run/ceph/ceph-mon.*.asok config show | grep
> > osd_crush_initial_weight
> > "osd_crush_initial_weight": "0.00",
> >
> > All looks good, now i am adding OSD with ceph-ansible and you know
> > what look like it don't honer that option and adding OSD with default
> > weight (In my case i have 1.9TB SSD so weight is 1.7)
> >
> > Can someone confirm injectargs work with osd_crush_initial_weight ?
> >
> >
> > Method: 2
> >
> > Now i have added that option in ceph-ansible playbook like following
> >
> > ceph_conf_overrides:
> >   osd:
> > osd_crush_initial_weight: 0
> >
> > and i run playbook and it did magic and added OSD with weight zero (0)
> >  but i have notice it restarted all OSD daemon on that node, i am
> > worried is it safe to restart osd daemon on ceph in production?
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Nautilus minor versions archive

[Paul Emmerich]

On Tue, Oct 1, 2019 at 11:21 PM Volodymyr Litovka  wrote:
>
> Dear colleagues,
>
> is archive of previos ceph minor versions  is available?

no, because reprepro doesn't support that

(yeah, that's clearly a solvable problem, but the current workflow
just uses reprepro and one repository)

Paul


>
> To be more specific - I'm looking for 14.2.1 or 14.2.2 for Ubuntu, but
> @download.ceph.com only 14.2.4 is available.
>
> Thank you.
>
> --
> Volodymyr Litovka
>"Vision without Execution is Hallucination." -- Thomas Edison
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Panic in kernel CephFS client after kernel update

[Ilya Dryomov]

On Tue, Oct 1, 2019 at 6:41 PM Kenneth Van Alstyne
 wrote:
>
> All:
> I’m not sure this should go to LKML or here, but I’ll start here.  After 
> upgrading from Linux kernel 4.19.60 to 4.19.75 (or 76), I started running 
> into kernel panics in the “ceph” module.  Based on the call trace, I believe 
> I was able to narrow it down to the following commit in the Linux kernel 4.19 
> source tree:
>
> commit 81281039a673d30f9d04d38659030a28051a
> Author: Yan, Zheng 
> Date:   Sun Jun 2 09:45:38 2019 +0800
>
> ceph: use ceph_evict_inode to cleanup inode's resource
>
> [ Upstream commit 87bc5b895d94a0f40fe170d4cf5771c8e8f85d15 ]
>
> remove_session_caps() relies on __wait_on_freeing_inode(), to wait for
> freeing inode to remove its caps. But VFS wakes freeing inode waiters
> before calling destroy_inode().
>
> Cc: sta...@vger.kernel.org
> Link: https://tracker.ceph.com/issues/40102
> Signed-off-by: "Yan, Zheng" 
> Reviewed-by: Jeff Layton 
> Signed-off-by: Ilya Dryomov 
> Signed-off-by: Sasha Levin 
>
>
> Backing this patch out and recompiling my kernel has since resolved my issues 
> (as far as I can tell thus far).  The issue was fairly easy to create by 
> simply creating and deleting files.  I tested using ‘dd’ and was pretty 
> consistently able to reproduce the issue. Since the issue occurred in a VM, I 
> do have a screenshot of the crashed machine and to avoid attaching an image, 
> I’ll link to where they are:  
> http://kvanals.kvanals.org/.ceph_kernel_panic_images/
>
> Am I way off base or has anyone else run into this issue?

Hi Kenneth,

This might be a botched backport.  The first version of this patch had
a conflict with Al's change that introduced ceph_free_inode() and Zheng
had to adjust it for that.  However, it looks like it has been taken to
4.19 verbatim, even though 4.19 does not have ceph_free_inode().

Zheng, Jeff, please take a look ASAP.

Thanks,

Ilya
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com