Oh BTW, I had to change back MTU to 1500 on the ceph-monitors because they
didn't work with 9000. This is the output of the ansible-playbook:
TASK [ceph-mon : put initial mon keyring in mon kv store]
**********************************************************************************************************************************************************************************
fatal: [ceph-monitor01]: FAILED! => {"changed": false, "cmd": ["ceph",
"--cluster", "ceph", "config-key", "put", "initial_mon_keyring",
"xxxxxxxxxxxxxxxxxxxxxxxxxx=="], "delta": "0:05:00.159094", "end":
"2017-10-30 09:48:10.425012", "failed": true, "msg": "non-zero return
code", "rc": 1, "start": "2017-10-30 09:43:10.265918", "stderr":
"2017-10-30 09:48:10.395156 7fd314408700 0 monclient(hunting):
authenticate timed out after 300\n2017-10-30 09:48:10.395197 7fd314408700
0 librados: client.admin authentication error (110) Connection timed
out\n[errno 110] error connecting to the cluster", "stderr_lines":
["2017-10-30 09:48:10.395156 7fd314408700 0 monclient(hunting):
authenticate timed out after 300", "2017-10-30 09:48:10.395197
7fd314408700 0 librados: client.admin authentication error (110)
Connection timed out", "[errno 110] error connecting to the cluster"],
"stdout": "", "stdout_lines": []}
Resuming, gateways and osds with jumbo frames, monitors not. Maybe this
isn't a problem because the servers that handle most of traffic are the
osds and gateways.
2017-10-30 10:50 GMT+01:00 Félix Barbeira <[email protected]>:
> Thanks Wido, it's fixed. I'm going to put the explanation if somebody runs
> into the same error.
>
> The MTU was defined on the client side and it was 9000. The 'ifconfig'
> shows the value established but if I ask directly the /proc filesystem it
> shows the following:
>
> root@ceph-node03:~# cat /proc/sys/net/ipv6/conf/eno1/mtu
> 1500
> root@ceph-node03:~#
>
> If I restart the interface it shows 9000 for a while and then it changes
> back to 1500. After some research it turns out that the router offers a MTU
> 1500 in the SLAAC parameters so when the session is 'refreshed', the client
> applies the wrong value (1500).
>
> The network guys changed the MTU parameter offered via SLAAC and now it's
> working:
>
> root@ceph-node03:~# cat /proc/sys/net/ipv6/conf/eno1/mtu
> 9000
> root@ceph-node03:~# ping6 -c 3 -M do -s 8952 ceph-node01
> PING ceph-node01(2a02:x:x:x:x:x:x:x) 8952 data bytes
> 8960 bytes from 2a02:x:x:x:x:x:x:x: icmp_seq=1 ttl=64 time=0.271 ms
> 8960 bytes from 2a02:x:x:x:x:x:x:x: icmp_seq=2 ttl=64 time=0.216 ms
> 8960 bytes from 2a02:x:x:x:x:x:x:x: icmp_seq=3 ttl=64 time=0.280 ms
>
> --- ceph-node01 ping statistics ---
> 3 packets transmitted, 3 received, 0% packet loss, time 2002ms
> rtt min/avg/max/mdev = 0.216/0.255/0.280/0.033 ms
> root@ceph-node03:~#
>
>
> 2017-10-27 16:02 GMT+02:00 Wido den Hollander <[email protected]>:
>
>>
>> > Op 27 oktober 2017 om 14:22 schreef Félix Barbeira <[email protected]
>> >:
>> >
>> >
>> > Hi,
>> >
>> > I'm trying to configure a ceph cluster using IPv6 only but I can't
>> enable
>> > jumbo frames. I made the definition on the
>> > 'interfaces' file and it seems like the value is applied but when I
>> test it
>> > looks like only works on IPv4, not IPv6.
>> >
>> > It works on IPv4:
>> >
>> > root@ceph-node01:~# ping -c 3 -M do -s 8972 ceph-node02
>> >
>> > PING ceph-node02 (x.x.x.x) 8972(9000) bytes of data.
>> > 8980 bytes from ceph-node02 (x.x.x.x): icmp_seq=1 ttl=64 time=0.474 ms
>> > 8980 bytes from ceph-node02 (x.x.x.x): icmp_seq=2 ttl=64 time=0.254 ms
>> > 8980 bytes from ceph-node02 (x.x.x.x): icmp_seq=3 ttl=64 time=0.288 ms
>> >
>>
>> Verify with Wireshark/tcpdump if it really sends 9k packets. I doubt it.
>>
>> > --- ceph-node02 ping statistics ---
>> > 3 packets transmitted, 3 received, 0% packet loss, time 2000ms
>> > rtt min/avg/max/mdev = 0.254/0.338/0.474/0.099 ms
>> >
>> > root@ceph-node01:~#
>> >
>> > But *not* in IPv6:
>> >
>> > root@ceph-node01:~# ping6 -c 3 -M do -s 8972 ceph-node02
>> > PING ceph-node02(x:x:x:x:x:x:x:x) 8972 data bytes
>> > ping: local error: Message too long, mtu=1500
>> > ping: local error: Message too long, mtu=1500
>> > ping: local error: Message too long, mtu=1500
>> >
>>
>> Like Ronny already mentioned, check the switches and the receiver. There
>> is a 1500 MTU somewhere configured.
>>
>> Wido
>>
>> > --- ceph-node02 ping statistics ---
>> > 4 packets transmitted, 0 received, +4 errors, 100% packet loss, time
>> 3024ms
>> >
>> > root@ceph-node01:~#
>> >
>> >
>> >
>> > root@ceph-node01:~# ifconfig
>> > eno1 Link encap:Ethernet HWaddr 24:6e:96:05:55:f8
>> > inet6 addr: 2a02:x:x:x:x:x:x:x/64 Scope:Global
>> > inet6 addr: fe80::266e:96ff:fe05:55f8/64 Scope:Link
>> > UP BROADCAST RUNNING MULTICAST *MTU:9000* Metric:1
>> > RX packets:633318 errors:0 dropped:0 overruns:0 frame:0
>> > TX packets:649607 errors:0 dropped:0 overruns:0 carrier:0
>> > collisions:0 txqueuelen:1000
>> > RX bytes:463355602 (463.3 MB) TX bytes:498891771 (498.8 MB)
>> >
>> > lo Link encap:Local Loopback
>> > inet addr:127.0.0.1 Mask:255.0.0.0
>> > inet6 addr: ::1/128 Scope:Host
>> > UP LOOPBACK RUNNING MTU:65536 Metric:1
>> > RX packets:127420 errors:0 dropped:0 overruns:0 frame:0
>> > TX packets:127420 errors:0 dropped:0 overruns:0 carrier:0
>> > collisions:0 txqueuelen:1
>> > RX bytes:179470326 (179.4 MB) TX bytes:179470326 (179.4 MB)
>> >
>> > root@ceph-node01:~#
>> >
>> > root@ceph-node01:~# cat /etc/network/interfaces
>> > # This file describes network interfaces avaiulable on your system
>> > # and how to activate them. For more information, see interfaces(5).
>> >
>> > source /etc/network/interfaces.d/*
>> >
>> > # The loopback network interface
>> > auto lo
>> > iface lo inet loopback
>> >
>> > # The primary network interface
>> > auto eno1
>> > iface eno1 inet6 auto
>> > post-up ifconfig eno1 mtu 9000
>> > root@ceph-node01:#
>> >
>> >
>> > Please help!
>> >
>> > --
>> > Félix Barbeira.
>> > _______________________________________________
>> > ceph-users mailing list
>> > [email protected]
>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>
>
>
> --
> Félix Barbeira.
>
--
Félix Barbeira.
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
