Hi GIanni, we get around 20Gbit/s over iperf with Kernel 5 with the untuned iperf command. Sorry, that i didn't mention that. Network seems to be completly fine. We don't have any packet drops or similar too.
Thanks, Alex Am 26.07.19 um 11:50 schrieb Gianni Milo: > Hi Alexander, > > Do you get same behaviour when using other tools like iperf for > example ? if so, then this might be not related to the DRBD itself, > but due to ethernet drivers issues for example.. > > Gianni > > > > > On Thu, 25 Jul 2019 at 13:26, Alexander Karamanlidis > <[email protected] > <mailto:[email protected]>> wrote: > > Hi everyone, > > > so we upgraded to PVE6 (Proxmox VE 6) about a week ago to test it. > > Since then our DRBD resource got incredibly slow (Around 105Mbit/s). > > If we boot to Kernel 4.15 our Speeds get back to normal (Max 15Gbit/s) > > > Here's some data: > > > root@node1:~# cat > /sys/kernel/debug/drbd/resources/r0/connections/node2/0/proc_drbd > ; echo > -e "\n\n" ; uname -a ; echo -e "\n\n" ; dpkg -l | grep > 'pve-kernel\|drbd' ; echo -e "\n\n" ; drbdadm dump > 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C > r----- > ns:0 nr:12242948 dw:12242948 dr:110085144 al:0 bm:0 lo:0 pe:[0;93] > ua:0 ap:[0;0] ep:1 wo:2 oos:7352525548 > [>....................] sync'ed: 1.5% (7180200/7287584)M > finish: 4:03:52 speed: 502,476 (530,020 -- 484,408) want: > 2,000,000 K/sec > 1% sector pos: 297469952/15002273264 > resync: used:2/61 hits:214057 misses:1684 starving:0 locked:0 > changed:842 > act_log: used:0/1237 hits:0 misses:0 starving:0 locked:0 > changed:0 > blocked on activity log: 0/0/0 > > > > Linux node1 4.15.18-18-pve #1 SMP PVE 4.15.18-44 (Wed, 03 Jul 2019 > 11:19:13 +0200) x86_64 GNU/Linux > > > > ii drbd-dkms > 9.0.19-1 > all RAID 1 over TCP/IP for Linux module source > ii drbd-utils > 9.10.0-1 > amd64 RAID 1 over TCP/IP for Linux (user utilities) > ii drbdtop > 0.2.1-1 > amd64 like top, but for drbd > ii pve-firmware > 3.0-2 > all Binary firmware code for the pve-kernel > ii pve-kernel-4.15 > 5.4-6 > all Latest Proxmox VE Kernel Image > ii pve-kernel-4.15.18-12-pve > 4.15.18-36 > amd64 The Proxmox PVE Kernel Image > ii pve-kernel-4.15.18-16-pve > 4.15.18-41 > amd64 The Proxmox PVE Kernel Image > ii pve-kernel-4.15.18-18-pve > 4.15.18-44 > amd64 The Proxmox PVE Kernel Image > ii pve-kernel-5.0 > 6.0-5 > all Latest Proxmox VE Kernel Image > ii pve-kernel-5.0.15-1-pve > 5.0.15-1 > amd64 The Proxmox PVE Kernel Image > ii pve-kernel-helper > 6.0-5 > all Function for various kernel maintenance tasks. > > > > # /etc/drbd.conf > # resource r0 on node1: not ignored, not stacked > # defined at /etc/drbd.d/r0.res:1 > resource r0 { > on node1 { > node-id 1; > volume 0 { > device /dev/drbd0 minor 0; > disk > /dev/disk/by-uuid/8a879a82-3880-4998-b5cb-70a95ce4bf79; > meta-disk internal; > } > address ipv4 192.168.99.1:7788 > <http://192.168.99.1:7788>; > } > on node2 { > node-id 0; > volume 0 { > device /dev/drbd0 minor 0; > disk > /dev/disk/by-uuid/8a879a82-3880-4998-b5cb-70a95ce4bf79; > meta-disk internal; > } > address ipv4 192.168.99.2:7788 > <http://192.168.99.2:7788>; > } > net { > after-sb-0pri discard-zero-changes; > after-sb-1pri discard-secondary; > after-sb-2pri disconnect; > csums-alg sha1; > max-buffers 36864; > max-epoch-size 20000; > rcvbuf-size 2097152; > sndbuf-size 1048576; > verify-alg sha1; > } > disk { > c-fill-target 10240; > c-max-rate 2237280; > c-min-rate 204800; > c-plan-ahead 0; > resync-rate 2000000; > } > } > > > > If we put I/O on the DRBD Resource we get a maximum of 14.19Gbit/s (We > have 25Gbit/s direct Attached Network) on our bond interface with the > 4.15 Kernel > > > > > > > root@node1:~# cat > /sys/kernel/debug/drbd/resources/r0/connections/node2/0/proc_drbd > ; echo > -e "\n\n" ; uname -a ; echo -e "\n\n" ; dpkg -l | grep > 'pve-kernel\|drbd' ; echo -e "\n\n" ; drbdadm dump > 0: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C > r----- > ns:0 nr:541700 dw:541700 dr:10270724 al:0 bm:0 lo:0 pe:[0;107] > ua:0 > ap:[0;0] ep:1 wo:2 oos:7334758124 > [>....................] sync'ed: 0.2% (7162848/7172768)M > finish: 14:05:20 speed: 144,596 (154,112 -- 166,556) want: > 2,000,000 K/sec > 2% sector pos: 332978176/15002273264 > resync: used:2/61 hits:19875 misses:162 starving:0 locked:0 > changed:81 > act_log: used:0/1237 hits:0 misses:0 starving:0 locked:0 > changed:0 > blocked on activity log: 0/0/0 > > > > Linux node1 5.0.15-1-pve #1 SMP PVE 5.0.15-1 (Wed, 03 Jul 2019 > 10:51:57 > +0200) x86_64 GNU/Linux > > > > ii drbd-dkms > 9.0.19-1 > all RAID 1 over TCP/IP for Linux module source > ii drbd-utils > 9.10.0-1 > amd64 RAID 1 over TCP/IP for Linux (user utilities) > ii drbdtop > 0.2.1-1 > amd64 like top, but for drbd > ii pve-firmware > 3.0-2 > all Binary firmware code for the pve-kernel > ii pve-kernel-4.15 > 5.4-6 > all Latest Proxmox VE Kernel Image > ii pve-kernel-4.15.18-12-pve > 4.15.18-36 > amd64 The Proxmox PVE Kernel Image > ii pve-kernel-4.15.18-16-pve > 4.15.18-41 > amd64 The Proxmox PVE Kernel Image > ii pve-kernel-4.15.18-18-pve > 4.15.18-44 > amd64 The Proxmox PVE Kernel Image > ii pve-kernel-5.0 > 6.0-5 > all Latest Proxmox VE Kernel Image > ii pve-kernel-5.0.15-1-pve > 5.0.15-1 > amd64 The Proxmox PVE Kernel Image > ii pve-kernel-helper > 6.0-5 > all Function for various kernel maintenance tasks. > > > > # /etc/drbd.conf > # resource r0 on node1: not ignored, not stacked > # defined at /etc/drbd.d/r0.res:1 > resource r0 { > on node1 { > node-id 1; > volume 0 { > device /dev/drbd0 minor 0; > disk > /dev/disk/by-uuid/8a879a82-3880-4998-b5cb-70a95ce4bf79; > meta-disk internal; > } > address ipv4 192.168.99.1:7788 > <http://192.168.99.1:7788>; > } > on node2 { > node-id 0; > volume 0 { > device /dev/drbd0 minor 0; > disk > /dev/disk/by-uuid/8a879a82-3880-4998-b5cb-70a95ce4bf79; > meta-disk internal; > } > address ipv4 192.168.99.2:7788 > <http://192.168.99.2:7788>; > } > net { > after-sb-0pri discard-zero-changes; > after-sb-1pri discard-secondary; > after-sb-2pri disconnect; > csums-alg sha1; > max-buffers 36864; > max-epoch-size 20000; > rcvbuf-size 2097152; > sndbuf-size 1048576; > verify-alg sha1; > } > disk { > c-fill-target 10240; > c-max-rate 2237280; > c-min-rate 204800; > c-plan-ahead 0; > resync-rate 2000000; > } > } > > If we put I/O on the DRBD Resource we get a maximum of 107Mbit/s (We > have 25Gbit/s direct Attached Network) on our bond interface with the > 5.0.15 Kernel > > > > Maybe someone has a clue what changed in Kernel 5 that is slowing us > that much down. > > Maybe someone even knows a solution for that. > > > > Kind Regards, > > Alexander Karamanlidis > > > > > > _______________________________________________ > Star us on GITHUB: https://github.com/LINBIT > drbd-user mailing list > [email protected] <mailto:[email protected]> > http://lists.linbit.com/mailman/listinfo/drbd-user > -- Freundliche Grüße Kind regards, Alexander Karamanlidis IT Systemadministrator Phone: +49 721 480 848 – 609 Lindenbaum GmbH Conferencing - Virtual Dialogues Facebook | LinkedIn | Youtube | Website Head office: Ludwig-Erhard-Allee 34 im Park Office, 76131 Karlsruhe Registration court: Amtsgericht Mannheim, HRB 706184 Managing director: Maarten Kronenburg Tax number: 35007/02060, USt. ID: DE 263797265 Lindenbaum auf der CallCenterWorld – und auf dem Mobile World Congress
_______________________________________________ Star us on GITHUB: https://github.com/LINBIT drbd-user mailing list [email protected] http://lists.linbit.com/mailman/listinfo/drbd-user
