Kefu has just pointed out that this has the hallmarks of
https://github.com/ceph/ceph/pull/13275
On Fri, Feb 24, 2017 at 3:00 PM, Brad Hubbard <bhubb...@redhat.com> wrote:
> Hmm,
>
> What's interesting is the feature set reported by the servers has only
> changed from
>
> e0106b84a846a42
>
> Bit 1 set Bit 6 set Bit 9 set Bit 11 set Bit 13 set Bit 14 set Bit 18
> set Bit 23 set Bit 25 set Bit 27 set Bit 30 set Bit 35 set Bit 36 set
> Bit 37 set Bit 39 set Bit 41 set Bit 42 set Bit 48 set Bit 57 set Bit
> 58 set Bit 59 set
>
> to
>
> e0106b84a846a52
>
> Bit 1 set Bit 4 set Bit 6 set Bit 9 set Bit 11 set Bit 13 set Bit 14
> set Bit 18 set Bit 23 set Bit 25 set Bit 27 set Bit 30 set Bit 35 set
> Bit 36 set Bit 37 set Bit 39 set Bit 41 set Bit 42 set Bit 48 set Bit
> 57 set Bit 58 set Bit 59 set
>
> So all it's done is *added* Bit 4 which is DEFINE_CEPH_FEATURE( 4, 1,
> SUBSCRIBE2)
>
>
> On Fri, Feb 24, 2017 at 1:40 PM, Schlacta, Christ <aarc...@aarcane.org> wrote:
>> # begin crush map
>> tunable choose_local_tries 0
>> tunable choose_local_fallback_tries 0
>> tunable choose_total_tries 50
>> tunable chooseleaf_descend_once 1
>> tunable chooseleaf_vary_r 1
>> tunable straw_calc_version 1
>> tunable allowed_bucket_algs 54
>>
>> # devices
>> device 0 osd.0
>> device 1 osd.1
>> device 2 osd.2
>>
>> # types
>> type 0 osd
>> type 1 host
>> type 2 chassis
>> type 3 rack
>> type 4 row
>> type 5 pdu
>> type 6 pod
>> type 7 room
>> type 8 datacenter
>> type 9 region
>> type 10 root
>>
>> # buckets
>> host densetsu {
>> id -2 # do not change unnecessarily
>> # weight 0.293
>> alg straw
>> hash 0 # rjenkins1
>> item osd.0 weight 0.146
>> item osd.1 weight 0.146
>> }
>> host density {
>> id -3 # do not change unnecessarily
>> # weight 0.145
>> alg straw
>> hash 0 # rjenkins1
>> item osd.2 weight 0.145
>> }
>> root default {
>> id -1 # do not change unnecessarily
>> # weight 0.438
>> alg straw
>> hash 0 # rjenkins1
>> item densetsu weight 0.293
>> item density weight 0.145
>> }
>>
>> # rules
>> rule replicated_ruleset {
>> ruleset 0
>> type replicated
>> min_size 1
>> max_size 10
>> step take default
>> step chooseleaf firstn 0 type host
>> step emit
>> }
>>
>> # end crush map
>>
>> On Thu, Feb 23, 2017 at 7:37 PM, Brad Hubbard <bhubb...@redhat.com> wrote:
>>> Did you dump out the crushmap and look?
>>>
>>> On Fri, Feb 24, 2017 at 1:36 PM, Schlacta, Christ <aarc...@aarcane.org>
>>> wrote:
>>>> insofar as I can tell, yes. Everything indicates that they are in effect.
>>>>
>>>> On Thu, Feb 23, 2017 at 7:14 PM, Brad Hubbard <bhubb...@redhat.com> wrote:
>>>>> Is your change reflected in the current crushmap?
>>>>>
>>>>> On Fri, Feb 24, 2017 at 12:07 PM, Schlacta, Christ <aarc...@aarcane.org>
>>>>> wrote:
>>>>>> ---------- Forwarded message ----------
>>>>>> From: Schlacta, Christ <aarc...@aarcane.org>
>>>>>> Date: Thu, Feb 23, 2017 at 6:06 PM
>>>>>> Subject: Re: [ceph-users] Upgrade Woes on suse leap with OBS ceph.
>>>>>> To: Brad Hubbard <bhubb...@redhat.com>
>>>>>>
>>>>>>
>>>>>> So setting the above to 0 by sheer brute force didn't work, so it's
>>>>>> not crush or osd problem.. also, the errors still say mon0, so I
>>>>>> suspect it's related to communication between libceph in kernel and
>>>>>> the mon.
>>>>>>
>>>>>> aarcane@densetsu:/etc/target$ sudo ceph --cluster rk osd crush tunables
>>>>>> hammer
>>>>>> adjusted tunables profile to hammer
>>>>>> aarcane@densetsu:/etc/target$ ceph --cluster rk osd crush show-tunables
>>>>>> {
>>>>>> "choose_local_tries": 0,
>>>>>> "choose_local_fallback_tries": 0,
>>>>>> "choose_total_tries": 50,
>>>>>> "chooseleaf_descend_once": 1,
>>>>>> "chooseleaf_vary_r": 1,
>>>>>> "chooseleaf_stable": 0,
>>>>>> "straw_calc_version": 1,
>>>>>> "allowed_bucket_algs": 54,
>>>>>> "profile": "hammer",
>>>>>> "optimal_tunables": 0,
>>>>>> "legacy_tunables": 0,
>>>>>> "minimum_required_version": "firefly",
>>>>>> "require_feature_tunables": 1,
>>>>>> "require_feature_tunables2": 1,
>>>>>> "has_v2_rules": 0,
>>>>>> "require_feature_tunables3": 1,
>>>>>> "has_v3_rules": 0,
>>>>>> "has_v4_buckets": 0,
>>>>>> "require_feature_tunables5": 0,
>>>>>> "has_v5_rules": 0
>>>>>> }
>>>>>>
>>>>>> aarcane@densetsu:/etc/target$ sudo rbd --cluster rk map rt1
>>>>>> rbd: sysfs write failed
>>>>>> In some cases useful info is found in syslog - try "dmesg | tail" or so.
>>>>>> rbd: map failed: (110) Connection timed out
>>>>>> aarcane@densetsu:~$ dmesg | tail
>>>>>> [10118.778868] libceph: mon0 10.0.0.67:6789 feature set mismatch, my
>>>>>> 40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000
>>>>>> [10118.779597] libceph: mon0 10.0.0.67:6789 missing required protocol
>>>>>> features
>>>>>> [10119.834634] libceph: mon0 10.0.0.67:6789 feature set mismatch, my
>>>>>> 40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000
>>>>>> [10119.835174] libceph: mon0 10.0.0.67:6789 missing required protocol
>>>>>> features
>>>>>> [10120.762983] libceph: mon0 10.0.0.67:6789 feature set mismatch, my
>>>>>> 40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000
>>>>>> [10120.763707] libceph: mon0 10.0.0.67:6789 missing required protocol
>>>>>> features
>>>>>> [10121.787128] libceph: mon0 10.0.0.67:6789 feature set mismatch, my
>>>>>> 40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000
>>>>>> [10121.787847] libceph: mon0 10.0.0.67:6789 missing required protocol
>>>>>> features
>>>>>> [10122.911117] libceph: mon0 10.0.0.67:6789 feature set mismatch, my
>>>>>> 40106b84a842a52 < server's e0106b84a846a52, missing a00000000004000
>>>>>> [10122.911872] libceph: mon0 10.0.0.67:6789 missing required protocol
>>>>>> features
>>>>>> aarcane@densetsu:~$
>>>>>>
>>>>>>
>>>>>> On Thu, Feb 23, 2017 at 5:56 PM, Schlacta, Christ <aarc...@aarcane.org>
>>>>>> wrote:
>>>>>>> They're from the suse leap ceph team. They maintain ceph, and build
>>>>>>> up to date versions for suse leap. What I don't know is how to
>>>>>>> disable it. When I try, I get the following mess:
>>>>>>>
>>>>>>> aarcane@densetsu:/etc/target$ ceph --cluster rk osd crush set-tunable
>>>>>>> require_feature_tunables5 0
>>>>>>> Invalid command: require_feature_tunables5 not in straw_calc_version
>>>>>>> osd crush set-tunable straw_calc_version <int> : set crush tunable
>>>>>>> <tunable> to <value>
>>>>>>> Error EINVAL: invalid command
>>>>>>>
>>>>>>> On Thu, Feb 23, 2017 at 5:54 PM, Brad Hubbard <bhubb...@redhat.com>
>>>>>>> wrote:
>>>>>>>> On Fri, Feb 24, 2017 at 11:00 AM, Schlacta, Christ
>>>>>>>> <aarc...@aarcane.org> wrote:
>>>>>>>>> aarcane@densetsu:~$ ceph --cluster rk osd crush show-tunables
>>>>>>>>> {
>>>>>>>>> "choose_local_tries": 0,
>>>>>>>>> "choose_local_fallback_tries": 0,
>>>>>>>>> "choose_total_tries": 50,
>>>>>>>>> "chooseleaf_descend_once": 1,
>>>>>>>>> "chooseleaf_vary_r": 1,
>>>>>>>>> "chooseleaf_stable": 1,
>>>>>>>>> "straw_calc_version": 1,
>>>>>>>>> "allowed_bucket_algs": 54,
>>>>>>>>> "profile": "jewel",
>>>>>>>>> "optimal_tunables": 1,
>>>>>>>>> "legacy_tunables": 0,
>>>>>>>>> "minimum_required_version": "jewel",
>>>>>>>>> "require_feature_tunables": 1,
>>>>>>>>> "require_feature_tunables2": 1,
>>>>>>>>> "has_v2_rules": 0,
>>>>>>>>> "require_feature_tunables3": 1,
>>>>>>>>> "has_v3_rules": 0,
>>>>>>>>> "has_v4_buckets": 0,
>>>>>>>>> "require_feature_tunables5": 1,
>>>>>>>>
>>>>>>>> I suspect setting the above to 0 would resolve the issue with the
>>>>>>>> client but there may be a reason why this is set?
>>>>>>>>
>>>>>>>> Where did those packages come from?
>>>>>>>>
>>>>>>>>> "has_v5_rules": 0
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> On Thu, Feb 23, 2017 at 4:45 PM, Brad Hubbard <bhubb...@redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>> On Thu, Feb 23, 2017 at 5:18 PM, Schlacta, Christ
>>>>>>>>>> <aarc...@aarcane.org> wrote:
>>>>>>>>>>> So I updated suse leap, and now I'm getting the following error from
>>>>>>>>>>> ceph. I know I need to disable some features, but I'm not sure what
>>>>>>>>>>> they are.. Looks like 14, 57, and 59, but I can't figure out what
>>>>>>>>>>> they correspond to, nor therefore, how to turn them off.
>>>>>>>>>>>
>>>>>>>>>>> libceph: mon0 10.0.0.67:6789 feature set mismatch, my
>>>>>>>>>>> 40106b84a842a42
>>>>>>>>>>> < server's e0106b84a846a42, missing a00000000004000
>>>>>>>>>>
>>>>>>>>>> http://cpp.sh/2rfy says...
>>>>>>>>>>
>>>>>>>>>> Bit 14 set
>>>>>>>>>> Bit 57 set
>>>>>>>>>> Bit 59 set
>>>>>>>>>>
>>>>>>>>>> Comparing this to
>>>>>>>>>> https://github.com/ceph/ceph/blob/master/src/include/ceph_features.h
>>>>>>>>>> shows...
>>>>>>>>>>
>>>>>>>>>> DEFINE_CEPH_FEATURE(14, 2, SERVER_KRAKEN)
>>>>>>>>>> DEFINE_CEPH_FEATURE(57, 1, MON_STATEFUL_SUB)
>>>>>>>>>> DEFINE_CEPH_FEATURE(57, 1, MON_ROUTE_OSDMAP) // overlap
>>>>>>>>>> DEFINE_CEPH_FEATURE(57, 1, OSDSUBOP_NO_SNAPCONTEXT) // overlap
>>>>>>>>>> DEFINE_CEPH_FEATURE(57, 1, SERVER_JEWEL) // overlap
>>>>>>>>>> DEFINE_CEPH_FEATURE(59, 1, FS_BTIME)
>>>>>>>>>> DEFINE_CEPH_FEATURE(59, 1, FS_CHANGE_ATTR) // overlap
>>>>>>>>>> DEFINE_CEPH_FEATURE(59, 1, MSG_ADDR2) // overlap
>>>>>>>>>>
>>>>>>>>>> $ echo "obase=16;ibase=16;$(echo e0106b84a846a42-a00000000004000|tr
>>>>>>>>>> '[a-z]' '[A-Z]')"|bc -qi
>>>>>>>>>> obase=16;ibase=16;E0106B84A846A42-A00000000004000
>>>>>>>>>> 40106B84A842A42
>>>>>>>>>>
>>>>>>>>>> So "me" (the client kernel) does not have the above features that are
>>>>>>>>>> present on the servers.
>>>>>>>>>>
>>>>>>>>>> Can you post the output of "ceph osd crush show-tunables"?
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> SuSE Leap 42.2 is Up to date as of tonight, no package updates
>>>>>>>>>>> available.
>>>>>>>>>>> All the ceph packages have the following version:
>>>>>>>>>>>
>>>>>>>>>>> 11.1.0+git.1486588482.ba197ae-72.1
>>>>>>>>>>>
>>>>>>>>>>> And the kernel has version:
>>>>>>>>>>>
>>>>>>>>>>> 4.4.49-16.1
>>>>>>>>>>>
>>>>>>>>>>> It was working perfectly before the upgrade.
>>>>>>>>>>>
>>>>>>>>>>> Thank you very much
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> ceph-users mailing list
>>>>>>>>>>> ceph-users@lists.ceph.com
>>>>>>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Cheers,
>>>>>>>>>> Brad
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Cheers,
>>>>>>>> Brad
>>>>>> _______________________________________________
>>>>>> ceph-users mailing list
>>>>>> ceph-users@lists.ceph.com
>>>>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> Brad
>>>
>>>
>>>
>>> --
>>> Cheers,
>>> Brad
>
>
>
> --
> Cheers,
> Brad
--
Cheers,
Brad
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
