Re: [ceph-users] iSCSI on Ubuntu and HA / Multipathing

[Frédéric Nass]

Hi Edward, 

What "Red Hat Enterprise Linux/CentOS 7.5 (or newer); Linux kernel v4.16 (or 
newer)" means is that you either need to use RHEL/CentOS 7.5 distribution with 
a 3.10.0-852+ kernel or any other distribution with a 4.16+ upstream kernel. 

Regards, 
Frédéric. 

- Le 10 Juil 19, à 22:34, Edward Kalk  a écrit : 

> The Docs say : [ http://docs.ceph.com/docs/nautilus/rbd/iscsi-targets/ |
> http://docs.ceph.com/docs/nautilus/rbd/iscsi-targets/ ]

> * Red Hat Enterprise Linux/CentOS 7.5 (or newer); Linux kernel v4.16 (or 
> newer)

> ^^Is there a version combination of CEPH and Ubuntu that works? Is anyone
> running iSCSI on Ubuntu ?
> -Ed

> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


smime.p7s
Description: S/MIME Cryptographic Signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] iSCSI on Ubuntu and HA / Multipathing

[Michael Christie]

On 07/11/2019 05:34 AM, Edward Kalk wrote:
> The Docs say : http://docs.ceph.com/docs/nautilus/rbd/iscsi-targets/
> 
>   * Red Hat Enterprise Linux/CentOS 7.5 (or newer); Linux kernel v4.16
> (or newer)
> 
> ^^Is there a version combination of CEPH and Ubuntu that works? Is
> anyone running iSCSI on Ubuntu ?
> -Ed
> 

There is still one rpm dep left. See attached hack patch.

We just need to make that kernel check generic and not use the rpm
labelCompare call. For RH/tcmu-runner based distros we do not need a
specific kernel version check plus the distro version check.
tcmu-runner/target_core_user should return an error if something is not
supported if the kernel is not supported.

Ricardo,

For SUSE/target_core_rbd, I think we just need to add a check for if
target_core_rbd is available, right? Maybe we want a backstore specific
callout/check?
diff --git a/rbd-target-api.py b/rbd-target-api.py
index 690a045..8dce0e8 100755
--- a/rbd-target-api.py
+++ b/rbd-target-api.py
@@ -16,7 +16,6 @@ import inspect
 import copy
 
 from functools import (reduce, wraps)
-from rpm import labelCompare
 import rados
 import rbd
 
@@ -2637,19 +2636,6 @@ def pre_reqs_errors():
 else:
 errors_found.append("OS is unsupported")
 
-# check the running kernel is OK (required kernel has patches to rbd.ko)
-os_info = os.uname()
-this_arch = os_info[-1]
-this_kernel = os_info[2].replace(".{}".format(this_arch), '')
-this_ver, this_rel = this_kernel.split('-', 1)
-
-# use labelCompare from the rpm module to handle the comparison
-if labelCompare(('1', this_ver, this_rel), ('1', k_vers, k_rel)) < 0:
-logger.error("Kernel version check failed")
-errors_found.append("Kernel version too old - {}-{} "
-"or above needed".format(k_vers,
- k_rel))
-
 return errors_found
 
 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ubuntu 18.04 - Mimic - Nautilus

[Harry G. Coin]

If you use the dashboard module and it's working now -- don't change 
because it's broken totally in Disco and later (ceph v13 as well).


On 7/10/19 2:51 PM, Reed Dier wrote:

It would appear by looking at the Ceph repo: https://download.ceph.com/
That Nautilus and Mimic are being built for Xenial and Bionic, where 
Luminous and Jewel are being built for Trusty and Xenial.


Then from Ubuntu in their main repos, they are publishing Jewel for 
Xenial, Luminous for Bionic, Mimic for Cosmic and Disco, and Nautilus 
for Eoan.

These packages are not updated at the bleeding edge mind you.

So there should be no issues with package availability between 
Nautilus and Xenial, however there should also be no impediment to 
moving from Xenial to Bionic as well.


I would expect to see Octopus (and likely P as well) target Bionic and 
whatever the 20.04 F release will be called.


Hope that clarifies things.

Reed

On Jul 10, 2019, at 2:36 PM, Edward Kalk > wrote:


Interesting. So is it not good that I am running Ubuntu 16.04 and 
14.2.1. ?

-Ed

On Jul 10, 2019, at 1:46 PM, Reed Dier > wrote:


It does not appear that that page has been updated in a while.

The official Ceph deb repos only include Mimic and Nautilus packages 
for 18.04,

While the Ubuntu-bionic repos include a Luminous build.

Hope that helps.

Reed

On Jul 10, 2019, at 1:20 PM, Edward Kalk > wrote:


When reviewing: 
http://docs.ceph.com/docs/master/start/os-recommendations/ I see 
there is no mention of “mimic” or “nautilus”.
What are the OS recommendations for them, specifically nautilus 
which is the one I’m running?


Is 18.04 advisable at all?

-Ed
___
ceph-users mailing list
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com







___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] iSCSI on Ubuntu and HA / Multipathing

[Edward Kalk]

The Docs say : http://docs.ceph.com/docs/nautilus/rbd/iscsi-targets/ 

Red Hat Enterprise Linux/CentOS 7.5 (or newer); Linux kernel v4.16 (or newer)
^^Is there a version combination of CEPH and Ubuntu that works? Is anyone 
running iSCSI on Ubuntu ?
-Ed___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ubuntu 18.04 - Mimic - Nautilus

[Kai Wagner]

On 10.07.19 20:46, Reed Dier wrote:
> It does not appear that that page has been updated in a while.

Addressed that already - someone needs to merge it

https://github.com/ceph/ceph/pull/28643

-- 
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 21284 (AG Nürnberg)




signature.asc
Description: OpenPGP digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Any news on dashboard regression / cython fix?

[Harry G. Coin]

Anyone know when the dashboard module will be working on any of the 
current and newer distros?


The dashboard module won't load / is totally broken on all distros 
shipping binaries with cython>0.29.   That dashboard module provides a 
big 'at a glance' help.


The 'workaround' that I saw posted a while ago attempts to avoid the bug 
report without changing the reason for it, and it appears to be 
available only for those that build from source owing to hackage.


misc detail for the curious:

uname -a
Linux ... 5.0.0-20-generic #21-Ubuntu SMP Mon Jun 24 09:32:09 UTC 2019 
x86_64 x86_64 x86_64 GNU/Linux


From ubuntu/disco:

# ceph -v
ceph version 13.2.4 (b10be4d44915a4d78a8e06aa31919e74927b142e) mimic 
(stable)


# ceph -s
cluster:
...
health: HEALTH_ERR
...

Module 'dashboard' has failed dependency: Interpreter change detected - 
this module can only be loaded into one interpreter per process.


apt list cython
Listing... Done
cython/disco 0.29.2-2build1 amd64



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ubuntu 18.04 - Mimic - Nautilus

[Reed Dier]

It would appear by looking at the Ceph repo: https://download.ceph.com/ 

That Nautilus and Mimic are being built for Xenial and Bionic, where Luminous 
and Jewel are being built for Trusty and Xenial.

Then from Ubuntu in their main repos, they are publishing Jewel for Xenial, 
Luminous for Bionic, Mimic for Cosmic and Disco, and Nautilus for Eoan.
These packages are not updated at the bleeding edge mind you.

So there should be no issues with package availability between Nautilus and 
Xenial, however there should also be no impediment to moving from Xenial to 
Bionic as well.

I would expect to see Octopus (and likely P as well) target Bionic and whatever 
the 20.04 F release will be called.

Hope that clarifies things.

Reed

> On Jul 10, 2019, at 2:36 PM, Edward Kalk  wrote:
> 
> Interesting. So is it not good that I am running Ubuntu 16.04 and 14.2.1. ?
> -Ed
> 
>> On Jul 10, 2019, at 1:46 PM, Reed Dier > > wrote:
>> 
>> It does not appear that that page has been updated in a while.
>> 
>> The official Ceph deb repos only include Mimic and Nautilus packages for 
>> 18.04,
>> While the Ubuntu-bionic repos include a Luminous build.
>> 
>> Hope that helps.
>> 
>> Reed
>> 
>>> On Jul 10, 2019, at 1:20 PM, Edward Kalk >> > wrote:
>>> 
>>> When reviewing: http://docs.ceph.com/docs/master/start/os-recommendations/ 
>>>  I see there is 
>>> no mention of “mimic” or “nautilus”.
>>> What are the OS recommendations for them, specifically nautilus which is 
>>> the one I’m running?
>>> 
>>> Is 18.04 advisable at all?
>>> 
>>> -Ed
>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com 
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
>>> 
>> 
> 



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ubuntu 18.04 - Mimic - Nautilus

[Edward Kalk]

Interesting. So is it not good that I am running Ubuntu 16.04 and 14.2.1. ?
-Ed

> On Jul 10, 2019, at 1:46 PM, Reed Dier  wrote:
> 
> It does not appear that that page has been updated in a while.
> 
> The official Ceph deb repos only include Mimic and Nautilus packages for 
> 18.04,
> While the Ubuntu-bionic repos include a Luminous build.
> 
> Hope that helps.
> 
> Reed
> 
>> On Jul 10, 2019, at 1:20 PM, Edward Kalk > > wrote:
>> 
>> When reviewing: http://docs.ceph.com/docs/master/start/os-recommendations/ 
>>  I see there is 
>> no mention of “mimic” or “nautilus”.
>> What are the OS recommendations for them, specifically nautilus which is the 
>> one I’m running?
>> 
>> Is 18.04 advisable at all?
>> 
>> -Ed
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com 
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 3 OSDs stopped and unable to restart

[ifedo...@suse.de]

You might want to try manual rocksdb compaction using ceph-kvstore-tool.. Sent from my Huawei tablet Original Message Subject: Re: [ceph-users] 3 OSDs stopped and unable to restartFrom: Brett Chancellor To: Igor Fedotov CC: Ceph Users Once backfilling finished, the cluster was super slow, most osd's were filled with heartbeat_map errors.  When an OSD restarts it causes a cascade of other osd's to follow suit and restart.. logs like..  -3> 2019-07-10 18:34:50.046 7f34abf5b700 -1 osd.69 1348581 get_health_metrics reporting 21 slow ops, oldest is osd_op(client.115295041.0:17575966 15.c37fa482 15.c37fa482 (undecoded) ack+ondisk+write+known_if_redirected e1348522)    -2> 2019-07-10 18:34:50.967 7f34acf5d700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f3493f2b700' had timed out after 90    -1> 2019-07-10 18:34:50.967 7f34acf5d700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7f3493f2b700' had suicide timed out after 150     0> 2019-07-10 18:34:51.025 7f3493f2b700 -1 *** Caught signal (Aborted) ** in thread 7f3493f2b700 thread_name:tp_osd_tp ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable) 1: (()+0xf5d0) [0x7f34b57c25d0] 2: (pread64()+0x33) [0x7f34b57c1f63] 3: (KernelDevice::read_random(unsigned long, unsigned long, char*, bool)+0x238) [0x55bfdae5a448] 4: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned long, char*)+0xca) [0x55bfdae1271a] 5: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x20) [0x55bfdae3b440] 6: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long, rocksdb::Slice*, char*) const+0x960) [0x55bfdb466ba0] 7: (rocksdb::BlockFetcher::ReadBlockContents()+0x3e7) [0x55bfdb420c27] 8: (()+0x11146a4) [0x55bfdb40d6a4] 9: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::FilePrefetchBuffer*, rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry*, bool, rocksdb::GetContext*)+0x2cc) [0x55bfdb40f63c] 10: (rocksdb::DataBlockIter* rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::DataBlockIter*, bool, bool, bool, rocksdb::GetContext*, rocksdb::Status, rocksdb::FilePrefetchBuffer*)+0x169) [0x55bfdb41cb29] 11: (rocksdb::BlockBasedTableIterator::InitDataBlock()+0xc8) [0x55bfdb41e588] 12: (rocksdb::BlockBasedTableIterator::FindKeyForward()+0x8d) [0x55bfdb41e89d] 13: (()+0x10adde9) [0x55bfdb3a6de9] 14: (rocksdb::MergingIterator::Next()+0x44) [0x55bfdb4357c4] 15: (rocksdb::DBIter::FindNextUserEntryInternal(bool, bool)+0x762) [0x55bfdb32a092] 16: (rocksdb::DBIter::Next()+0x1d6) [0x55bfdb32b6c6] 17: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x2d) [0x55bfdad9fa8d] 18: (BlueStore::_collection_list(BlueStore::Collection*, ghobject_t const&, ghobject_t const&, int, std::vector >*, ghobject_t*)+0xdf6) [0x55bfdad12466] 19: (BlueStore::collection_list(boost::intrusive_ptr&, ghobject_t const&, ghobject_t const&, int, std::vector >*, ghobject_t*)+0x9b) [0x55bfdad1393b] 20: (PG::_delete_some(ObjectStore::Transaction*)+0x1e0) [0x55bfda984120] 21: (PG::RecoveryState::Deleting::react(PG::DeleteSome const&)+0x38) [0x55bfda985598] 22: (boost::statechart::simple_state, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x16a) [0x55bfda9c45ca] 23: (boost::statechart::state_machine, boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base const&)+0x5a) [0x55bfda9a20ca] 24: (PG::do_peering_event(std::shared_ptr, PG::RecoveryCtx*)+0x119) [0x55bfda991389] 25: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr, ThreadPool::TPHandle&)+0x1b4) [0x55bfda8cb3c4] 26: (OSD::dequeue_delete(OSDShard*, PG*, unsigned int, ThreadPool::TPHandle&)+0x234) [0x55bfda8cb804] 27: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x9f4) [0x55bfda8bfb44] 28: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x433) [0x55bfdaeb9e93] 29: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55bfdaebcf30] 30: (()+0x7dd5) [0x7f34b57badd5] 31: (clone()+0x6d) [0x7f34b4680ead] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this.--- logging levels ---   0/ 5 none   0/ 1 lockdep   0/ 1 context   1/ 1 crush   1/ 5 mds   1/ 5 mds_balancer   1/ 5 mds_locker   1/ 5 mds_log   1/ 5 mds_log_expire   1/ 5 mds_migrator   0/ 1 buffer   0/ 1 timer   0/ 1 filer   0/ 1 striper   0/ 1 objecter   0/ 5 rados   0/ 5 rbd   0/ 5 rbd_mirror   0/ 5 rbd_replay   0/ 5 journaler   0/ 5 objectcacher   0/ 5 client   1/ 5 osd   0/ 5 optracker   0/ 5 objclass   1/ 3 filestore   1/ 3 journal   0/ 0 ms   1/ 5 mon   0/10 monc   1/ 5 paxos   0/ 5 tp   1/ 5 auth   1/ 5 crypto   1/ 1 finisher   1/ 1 reserver   1/ 5 heartbeatmap   1/ 5 perfcounter   1/ 5 rgw   1/ 5 rgw_sync   1/10 civetweb   1/ 5 javaclient   1/ 5 

Re: [ceph-users] Ubuntu 18.04 - Mimic - Nautilus

[Reed Dier]

It does not appear that that page has been updated in a while.

The official Ceph deb repos only include Mimic and Nautilus packages for 18.04,
While the Ubuntu-bionic repos include a Luminous build.

Hope that helps.

Reed

> On Jul 10, 2019, at 1:20 PM, Edward Kalk  wrote:
> 
> When reviewing: http://docs.ceph.com/docs/master/start/os-recommendations/ 
>  I see there is 
> no mention of “mimic” or “nautilus”.
> What are the OS recommendations for them, specifically nautilus which is the 
> one I’m running?
> 
> Is 18.04 advisable at all?
> 
> -Ed
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



smime.p7s
Description: S/MIME cryptographic signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] 3 OSDs stopped and unable to restart

[Brett Chancellor]

Once backfilling finished, the cluster was super slow, most osd's were
filled with heartbeat_map errors.  When an OSD restarts it causes a cascade
of other osd's to follow suit and restart.. logs like..
  -3> 2019-07-10 18:34:50.046 7f34abf5b700 -1 osd.69 1348581
get_health_metrics reporting 21 slow ops, oldest is
osd_op(client.115295041.0:17575966 15.c37fa482 15.c37fa482 (undecoded)
ack+ondisk+write+known_if_redirected e1348522)
-2> 2019-07-10 18:34:50.967 7f34acf5d700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f3493f2b700' had timed out after 90
-1> 2019-07-10 18:34:50.967 7f34acf5d700  1 heartbeat_map is_healthy
'OSD::osd_op_tp thread 0x7f3493f2b700' had suicide timed out after 150
 0> 2019-07-10 18:34:51.025 7f3493f2b700 -1 *** Caught signal (Aborted)
**
 in thread 7f3493f2b700 thread_name:tp_osd_tp

 ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus
(stable)
 1: (()+0xf5d0) [0x7f34b57c25d0]
 2: (pread64()+0x33) [0x7f34b57c1f63]
 3: (KernelDevice::read_random(unsigned long, unsigned long, char*,
bool)+0x238) [0x55bfdae5a448]
 4: (BlueFS::_read_random(BlueFS::FileReader*, unsigned long, unsigned
long, char*)+0xca) [0x55bfdae1271a]
 5: (BlueRocksRandomAccessFile::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x20) [0x55bfdae3b440]
 6: (rocksdb::RandomAccessFileReader::Read(unsigned long, unsigned long,
rocksdb::Slice*, char*) const+0x960) [0x55bfdb466ba0]
 7: (rocksdb::BlockFetcher::ReadBlockContents()+0x3e7) [0x55bfdb420c27]
 8: (()+0x11146a4) [0x55bfdb40d6a4]
 9:
(rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::FilePrefetchBuffer*,
rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&,
rocksdb::BlockHandle const&, rocksdb::Slice,
rocksdb::BlockBasedTable::CachableEntry*, bool,
rocksdb::GetContext*)+0x2cc) [0x55bfdb40f63c]
 10: (rocksdb::DataBlockIter*
rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*,
rocksdb::ReadOptions const&, rocksdb::BlockHandle const&,
rocksdb::DataBlockIter*, bool, bool, bool, rocksdb::GetContext*,
rocksdb::Status, rocksdb::FilePrefetchBuffer*)+0x169) [0x55bfdb41cb29]
 11: (rocksdb::BlockBasedTableIterator::InitDataBlock()+0xc8) [0x55bfdb41e588]
 12: (rocksdb::BlockBasedTableIterator::FindKeyForward()+0x8d) [0x55bfdb41e89d]
 13: (()+0x10adde9) [0x55bfdb3a6de9]
 14: (rocksdb::MergingIterator::Next()+0x44) [0x55bfdb4357c4]
 15: (rocksdb::DBIter::FindNextUserEntryInternal(bool, bool)+0x762)
[0x55bfdb32a092]
 16: (rocksdb::DBIter::Next()+0x1d6) [0x55bfdb32b6c6]
 17: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::next()+0x2d)
[0x55bfdad9fa8d]
 18: (BlueStore::_collection_list(BlueStore::Collection*, ghobject_t
const&, ghobject_t const&, int, std::vector >*, ghobject_t*)+0xdf6) [0x55bfdad12466]
 19:
(BlueStore::collection_list(boost::intrusive_ptr&,
ghobject_t const&, ghobject_t const&, int, std::vector >*, ghobject_t*)+0x9b) [0x55bfdad1393b]
 20: (PG::_delete_some(ObjectStore::Transaction*)+0x1e0) [0x55bfda984120]
 21: (PG::RecoveryState::Deleting::react(PG::DeleteSome const&)+0x38)
[0x55bfda985598]
 22: (boost::statechart::simple_state,
(boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base
const&, void const*)+0x16a) [0x55bfda9c45ca]
 23: (boost::statechart::state_machine,
boost::statechart::null_exception_translator>::process_event(boost::statechart::event_base
const&)+0x5a) [0x55bfda9a20ca]
 24: (PG::do_peering_event(std::shared_ptr,
PG::RecoveryCtx*)+0x119) [0x55bfda991389]
 25: (OSD::dequeue_peering_evt(OSDShard*, PG*,
std::shared_ptr, ThreadPool::TPHandle&)+0x1b4)
[0x55bfda8cb3c4]
 26: (OSD::dequeue_delete(OSDShard*, PG*, unsigned int,
ThreadPool::TPHandle&)+0x234) [0x55bfda8cb804]
 27: (OSD::ShardedOpWQ::_process(unsigned int,
ceph::heartbeat_handle_d*)+0x9f4) [0x55bfda8bfb44]
 28: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x433)
[0x55bfdaeb9e93]
 29: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55bfdaebcf30]
 30: (()+0x7dd5) [0x7f34b57badd5]
 31: (clone()+0x6d) [0x7f34b4680ead]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed
to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 0 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 

[ceph-users] Ubuntu 18.04 - Mimic - Nautilus

[Edward Kalk]

When reviewing: http://docs.ceph.com/docs/master/start/os-recommendations/ 
 I see there is no 
mention of “mimic” or “nautilus”.
What are the OS recommendations for them, specifically nautilus which is the 
one I’m running?

Is 18.04 advisable at all?

-Ed___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Few OSDs crash on partner nodes when a node is rebooted

[Edward Kalk]

Wondering if the fix for http://tracker.ceph.com/issues/38724 
 will be included in : 
https://tracker.ceph.com/projects/ceph/roadmap#v14.2.2 
 ?

> On Jul 10, 2019, at 7:55 AM, Edward Kalk  wrote:
> 
> Hello,
> 
> Has anyone else seen this? Basically I reboot a node and 2-3 OSDs on other 
> hosts crash.
> Then they fail to start back up and have seem to hit a startup bug. 
> http://tracker.ceph.com/issues/38724 
> 
> What’s weird is that it seemed to be the same OSDs that crash, so as a work 
> around I tried to rip and rebuild them.
> Will test rebooting a node today and see if they still crash or if any others 
> crash this time.
> 
> -Ed Kalk
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Few OSDs crash on partner nodes when a node is rebooted

[Edward Kalk]

Hello,

Has anyone else seen this? Basically I reboot a node and 2-3 OSDs on other 
hosts crash.
Then they fail to start back up and have seem to hit a startup bug. 
http://tracker.ceph.com/issues/38724 

What’s weird is that it seemed to be the same OSDs that crash, so as a work 
around I tried to rip and rebuild them.
Will test rebooting a node today and see if they still crash or if any others 
crash this time.

-Ed Kalk___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] How does monitor know OSD is dead?

[Nathan Cutler]

I don't know if it's relevant here, but I saw similar behavior while 
implementing
a Luminous->Nautilus automated upgrade test. When I used a single-node cluster
with 4 OSDs, the Nautilus cluster would not function properly after the reboot.
IIRC some OSDs were reported by "ceph -s" as up, even though they weren't 
running.

I "fixed" the issue by adding a second node to the cluster. With two nodes (8
OSDs), the upgrade works fine.

I will reproduce the issue again and open a bug report.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] writable snapshots in cephfs? GDPR/DSGVO

[Yan, Zheng]

On Wed, Jul 10, 2019 at 4:16 PM Lars Täuber  wrote:
>
> Hi everbody!
>
> Is it possible to make snapshots in cephfs writable?
> We need to remove files because of this General Data Protection Regulation 
> also from snapshots.
>

It's possible (only delete data), but need to modify both mds and osd.
It's a big project.


> Thanks and best regards,
> Lars
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] writable snapshots in cephfs? GDPR/DSGVO

[Wido den Hollander]

On 7/10/19 9:59 AM, Lars Täuber wrote:
> Hi everbody!
> 
> Is it possible to make snapshots in cephfs writable?

As far as I'm aware: No

You would need to remove the complete snapshot and create a new one.

> We need to remove files because of this General Data Protection Regulation 
> also from snapshots.
> 

This goes outside the scope of Ceph, but GDPR doesn't require you to
remove the data instantly. Otherwise you also need to burn your tape
drives (should you have those) as long as you state how long you will
retain data for backup purposes.

Your use-case might be different though :-)

Oh, I'm not a lawyer either.

Wido

> Thanks and best regards,
> Lars
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic

[Alexandre DERUMIER]

> Can't say anything about latency.

>>Anybody else? Wido?

I'm running it on mimic since 1 month, no problem until now, and it's 
definility fixing the latency increasing over time. (aka need restart osd each 
week)

Memory usage is almost the same than before.


- Mail original -
De: "Konstantin Shalygin" 
À: "Marc Roos" , "Wido den Hollander" 
Cc: "ceph-users" 
Envoyé: Mercredi 10 Juillet 2019 05:56:35
Objet: Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic

On 5/28/19 5:16 PM, Marc Roos wrote: 
> I switched first of may, and did not notice to much difference in memory 
> usage. After the restart of the osd's on the node I see the memory 
> consumption gradually getting back to as before. 
> Can't say anything about latency. 


Anybody else? Wido? 

I see many patches from Igor comes to Luminous. And also bitmap 
allocator (default in Nautilus) is tries to kill Brett Chancellor 
cluster for a week [1] 



[1] 
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/035726.html 

k 

___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph features and linux kernel version for upmap

[Lars Marowsky-Bree]

On 2019-07-09T16:32:22, Mattia Belluco  wrote:

> That of course in my case fails with:
> Error EPERM: cannot set require_min_compat_client to luminous: 29
> connected client(s) look like jewel (missing 0x800); 1
> connected client(s) look like jewel (missing 0x800); add
> --yes-i-really-mean-it to do it anyway

That's because instead of looking at the actual feature flags needed and
advertised by the clients, it tries to translate those back to a Ceph
version (which doesn't work, because it's not the same exact codestream)
and then compares that one. That is so harebrained it probably deserves
a tracker bug ;-)



Regards,
Lars

-- 
SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG 
Nürnberg)
"Architects should open possibilities and not determine everything." (Ueli 
Zbinden)
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] writable snapshots in cephfs? GDPR/DSGVO

[Lars Täuber]

Hi everbody!

Is it possible to make snapshots in cephfs writable?
We need to remove files because of this General Data Protection Regulation also 
from snapshots.

Thanks and best regards,
Lars
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] BlueStore bitmap allocator under Luminous and Mimic

[Wido den Hollander]

On 7/10/19 5:56 AM, Konstantin Shalygin wrote:
> On 5/28/19 5:16 PM, Marc Roos wrote:
>> I switched first of may, and did not notice to much difference in memory
>> usage. After the restart of the osd's on the node I see the memory
>> consumption gradually getting back to as before.
>> Can't say anything about latency.
> 
> 
> Anybody else? Wido?
> 

I haven't tried it yet as I don't have a cluster running with Luminous I
can test this on.

So right now I've been sticking with Stupidallocator under Luminous.

> I see many patches from Igor comes to Luminous. And also bitmap
> allocator (default in Nautilus) is tries to kill Brett Chancellor
> cluster for a week [1]
> 
> 
> 
> [1]
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-July/035726.html
> 
> k
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com