Re: [Bacula-users] fd lose connection
Going further, I check my remote machine provider and it seems that there tcp services monitoring was the real problem for my brutal connection termination So they won't now perturbate my connections. I too activated tcp_bbr and sfq for queuing, and I reactivated psk passphrase and tls verify peer. I now have a stable backup So, sorry for all the issues I mentioned, which seem to be coming from the cloud services provider that has apparently been the source of these issues Be aware too that /sys/class/net/enp1s0/power/control is "auto" by default in Debian 12 and should be set to "on" to prevent device from being power managed during backup (even in linux ; ) (my eth device is not IEEE802.3az compliant but it could be a problem if it is) thanks -Message d'origine- De : Josh Fisher via Bacula-users Envoyé : jeudi 9 novembre 2023 01:16 À : Martin Simmons Cc : bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] fd lose connection On 11/8/23 13:32, Martin Simmons wrote: >>>>>> On Wed, 8 Nov 2023 11:09:44 -0500, Josh Fisher via Bacula-users said: >> On 11/7/23 19:26, Lionel PLASSE wrote: >>> I’m sorry but no wi-fi nor 5G in our factory, and don't use my phone too to >>> backup my servers :) . >>> I was talking about ssh (scp) transfer just to Just to show out I have no >>> problem when uploading big continuous data using other tools through this >>> wan. The wan connection is quite stable. >>> >>> "So it is fine when the NIC is up. Since this is Windows," >>> no windows. I discarder windows problem hypothesis by using a >>> migration job, so from linux sd to linux sd >> OK. I see that now. You also tried without compression and without >> encryption. Have you tried reducing Maximum Network Buffer Size back >> to the default 32768? > Are you sure it is 32768? > > I thought the default comes from this in bacula/src/baconfig.h: > > #define DEFAULT_NETWORK_BUFFER_SIZE (64 * 1024) In the docs it says the default is 32768, but if it's in the source, then that's what it is. :) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users smime.p7s Description: S/MIME cryptographic signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] fd lose connection
On 11/8/23 13:32, Martin Simmons wrote: On Wed, 8 Nov 2023 11:09:44 -0500, Josh Fisher via Bacula-users said: On 11/7/23 19:26, Lionel PLASSE wrote: I’m sorry but no wi-fi nor 5G in our factory, and don't use my phone too to backup my servers :) . I was talking about ssh (scp) transfer just to Just to show out I have no problem when uploading big continuous data using other tools through this wan. The wan connection is quite stable. "So it is fine when the NIC is up. Since this is Windows," no windows. I discarder windows problem hypothesis by using a migration job, so from linux sd to linux sd OK. I see that now. You also tried without compression and without encryption. Have you tried reducing Maximum Network Buffer Size back to the default 32768? Are you sure it is 32768? I thought the default comes from this in bacula/src/baconfig.h: #define DEFAULT_NETWORK_BUFFER_SIZE (64 * 1024) In the docs it says the default is 32768, but if it's in the source, then that's what it is. :) ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] fd lose connection
Ok , Without encryption (TlsPskEnable = false and TlsVerifyPeer=false) and lzo-md5 the job finished well and all job that were in error gone ok too today. Just to say , on Debian 12 , libssl is on the new v3.0 version so I have to retroport it from the bullseye repo in order to install Bacula which depends on libssl1.1 Perhaps it could be why I encounter this problem. (for info too, the ethernet card is : > OK. I see that now. You also tried without compression and without > encryption. Have you tried reducing Maximum Network Buffer Size back > to the default 32768? I also have tested with 32768 max net buffer size, but it was from window's fd to Debian 12 remote sd and it went in error (but really not sure if at this time all my confs were properly set in both side) : Error: lib/bsock.c:397 Wrote 32772 bytes to Storage daemon:192.168.0.18:9103, but only 32768 accepted. I now use without encr (and with lzo-md5 comp for data), Its ok, I can work like this. As the last past jobs are now ok I won't go further. Thank you for considering my problem. Great thanks for the Bacula and bbr explanation I will study this with great attention. Sure it could give me much more efficience. -Message d'origine- De : Martin Simmons Envoyé : mercredi 8 novembre 2023 19:32 À : Josh Fisher Cc : bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] fd lose connection >>>>> On Wed, 8 Nov 2023 11:09:44 -0500, Josh Fisher via Bacula-users said: > > On 11/7/23 19:26, Lionel PLASSE wrote: > > I’m sorry but no wi-fi nor 5G in our factory, and don't use my phone too to > > backup my servers :) . > > I was talking about ssh (scp) transfer just to Just to show out I have no > > problem when uploading big continuous data using other tools through this > > wan. The wan connection is quite stable. > > > > "So it is fine when the NIC is up. Since this is Windows," > > no windows. I discarder windows problem hypothesis by using a > > migration job, so from linux sd to linux sd > > OK. I see that now. You also tried without compression and without > encryption. Have you tried reducing Maximum Network Buffer Size back > to the default 32768? Are you sure it is 32768? I thought the default comes from this in bacula/src/baconfig.h: #define DEFAULT_NETWORK_BUFFER_SIZE (64 * 1024) >There must be some reason why the client seems to > be sending 30 bytes more than its Maximum Network Buffer Size. Bacula > first tries the Maximum Network Buffer Size, but if the OS does not > accept that size, then it adjusts the value down until the OS accepts > it. Maybe the actual buffer size gets calculated differently on Debian > 12? Why is the send size exceeding the buffer size? Or could there be > a typo in the Maximum Network Buffer Size setting on one side? It's an interesting question. OTOH, if the connection is using TLS, then the underlying number of bytes transmitted doesn't match the application layer anyway. Also, can you explain why the network buffer size would cause data loss on a TCP connection? > > > > > > Thanks for all, I will find out a solution Best regards > > > > PLASSE Lionel | Networks & Systems Administrator > > 221 Allée de Fétan > > 01600 TREVOUX - FRANCE > > Tel : +33(0)4.37.49.91.39 > > pla...@cofiem.fr > > www.cofiem.fr | www.cofiem-robotics.fr > > > > > > > > > > > > > > > > -Message d'origine- > > De : Josh Fisher via Bacula-users > > > > Envoyé : mardi 7 novembre 2023 18:01 À : > > bacula-users@lists.sourceforge.net > > Objet : Re: [Bacula-users] fd lose connection > > > > > > On 11/7/23 04:34, Lionel PLASSE wrote: > >> Hello , > >> > >> Could Encryption have any impact in my problem. > >> > >> I am testing without any encryption between SD/DIR/BConsole or FD > >> and it seems to be more stable. (short sized job right done , > >> longest job already running : 750Gb to migrate) > >> > >> My WAN connection seems to be quite good, I achieve transferring big > >> and small raw files by scp ssh and don't have ping latency or troubles > >> with the ipsec connection. > > > > So it is fine when the NIC is up. Since this is Windows, the first thing to > > do is turn off power saving for the network interface device in Device > > Manager. Make sure that the NIC doesn't ever power down its PHY. > > If any switch, router, or VPN doesn't handle energy-efficient internet in > > the same way, then it can look like a dropped connection to the other side. > > > > Also, you don't s
Re: [Bacula-users] fd lose connection
>>>>> On Wed, 8 Nov 2023 11:09:44 -0500, Josh Fisher via Bacula-users said: > > On 11/7/23 19:26, Lionel PLASSE wrote: > > I’m sorry but no wi-fi nor 5G in our factory, and don't use my phone too to > > backup my servers :) . > > I was talking about ssh (scp) transfer just to Just to show out I have no > > problem when uploading big continuous data using other tools through this > > wan. The wan connection is quite stable. > > > > "So it is fine when the NIC is up. Since this is Windows," > > no windows. I discarder windows problem hypothesis by using a migration > > job, so from linux sd to linux sd > > OK. I see that now. You also tried without compression and without > encryption. Have you tried reducing Maximum Network Buffer Size back to > the default 32768? Are you sure it is 32768? I thought the default comes from this in bacula/src/baconfig.h: #define DEFAULT_NETWORK_BUFFER_SIZE (64 * 1024) >There must be some reason why the client seems to be > sending 30 bytes more than its Maximum Network Buffer Size. Bacula first > tries the Maximum Network Buffer Size, but if the OS does not accept > that size, then it adjusts the value down until the OS accepts it. Maybe > the actual buffer size gets calculated differently on Debian 12? Why is > the send size exceeding the buffer size? Or could there be a typo in the > Maximum Network Buffer Size setting on one side? It's an interesting question. OTOH, if the connection is using TLS, then the underlying number of bytes transmitted doesn't match the application layer anyway. Also, can you explain why the network buffer size would cause data loss on a TCP connection? > > > > > > Thanks for all, I will find out a solution > > Best regards > > > > PLASSE Lionel | Networks & Systems Administrator > > 221 Allée de Fétan > > 01600 TREVOUX - FRANCE > > Tel : +33(0)4.37.49.91.39 > > pla...@cofiem.fr > > www.cofiem.fr | www.cofiem-robotics.fr > > > > > > > > > > > > > > > > -Message d'origine- > > De : Josh Fisher via Bacula-users > > Envoyé : mardi 7 novembre 2023 18:01 > > À : bacula-users@lists.sourceforge.net > > Objet : Re: [Bacula-users] fd lose connection > > > > > > On 11/7/23 04:34, Lionel PLASSE wrote: > >> Hello , > >> > >> Could Encryption have any impact in my problem. > >> > >> I am testing without any encryption between SD/DIR/BConsole or FD and > >> it seems to be more stable. (short sized job right done , longest job > >> already running : 750Gb to migrate) > >> > >> My WAN connection seems to be quite good, I achieve transferring big > >> and small raw files by scp ssh and don't have ping latency or troubles > >> with the ipsec connection. > > > > So it is fine when the NIC is up. Since this is Windows, the first thing to > > do is turn off power saving for the network interface device in Device > > Manager. Make sure that the NIC doesn't ever power down its PHY. > > If any switch, router, or VPN doesn't handle energy-efficient internet in > > the same way, then it can look like a dropped connection to the other side. > > > > Also, you don't say what type of WAN connection this is. Many wireless > > services, 5G, etc. can and will drop open sockets due to inactivity (or > > perceived inactivity) to free up channels. > > > > > >> I tried too with NAT, by not using IPSEC and setting Bacula SD & DIR > >> directly in front of the WAN. > >> And the same occurs (wrote X byte but only Y accepted) > >> > >> I Tried too to make a migration job to migrate from SD to SD through WAN > >> instead of SD-> FD through WAN and the result was the same. (to see if > >> win32 FD could be involved) > >>- DIR and SD in the same LAN. > >>- Backup remote FD through remote SD, the two are in the same LAN to > >> fast backup : step OK . > >> - Then migration from remote SD to the SD that is in the DIR area > >> through WAN to outsource volumes physical support : step nok The final > >> goal: outsourcing volumes. > >> I then discard the gzip compression (just in case) > >> > >> The errors are quite disturbing : > >> * Error: lib/bsock.c:397 Wrote 65566 bytes to > >> client:192.168.0.4:9102, but only 65536 accepted > >> Fatal error: filed/backup.c:1008 Network send error to SD. > >> ERR=Input/output error Or (when increasing MaximumNetworkBuf
Re: [Bacula-users] fd lose connection
Hello Lionel, On 11/7/23 19:26, Lionel PLASSE wrote: > I’m sorry but no wi-fi nor 5G in our factory, and don't use my phone too to > backup my servers :) . > I was talking about ssh (scp) transfer just to Just to show out I have no > problem when uploading big continuous data using other tools through this > wan. The wan connection is quite stable. > > "So it is fine when the NIC is up. Since this is Windows," > no windows. I discarder windows problem hypothesis by using a migration job, > so from linux sd to linux sd "OK. I see that now. You also tried without compression and without encryption. Have you tried reducing Maximum Network Buffer Size back to the default 32768? There must be some reason why the client seems to be sending 30 bytes more than its Maximum Network Buffer Size. Bacula first tries the Maximum Network Buffer Size, but if the OS does not accept that size, then it adjusts the value down until the OS accepts it. Maybe the actual buffer size gets calculated differently on Debian 12? Why is the send size exceeding the buffer size? Or could there be a typo in the Maximum Network Buffer Size setting on one side?" In addition to the Josh considerations, I would consider trying BBR: https://www.bacula.lat/bacula-e-prototoclo-bbr-para-backups-via-internet-perdas-de-pacote-desconecoes-etc/?lang=en I have a mixed OnP+cloud customer +1000 machines situations, where this was the only solution for the network interruptions. The BBR is not only more resilient to transmission contingencies (NATs, firewall drops, switches/router resets etc.), but also provides much faster throughput recovery in case of failure. Underlying VPN is also something that, by my experience, are prone to disconnections that have huge impact on Linux default congestion protocol. Rgds.___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] fd lose connection
On 11/7/23 19:26, Lionel PLASSE wrote: I’m sorry but no wi-fi nor 5G in our factory, and don't use my phone too to backup my servers :) . I was talking about ssh (scp) transfer just to Just to show out I have no problem when uploading big continuous data using other tools through this wan. The wan connection is quite stable. "So it is fine when the NIC is up. Since this is Windows," no windows. I discarder windows problem hypothesis by using a migration job, so from linux sd to linux sd OK. I see that now. You also tried without compression and without encryption. Have you tried reducing Maximum Network Buffer Size back to the default 32768? There must be some reason why the client seems to be sending 30 bytes more than its Maximum Network Buffer Size. Bacula first tries the Maximum Network Buffer Size, but if the OS does not accept that size, then it adjusts the value down until the OS accepts it. Maybe the actual buffer size gets calculated differently on Debian 12? Why is the send size exceeding the buffer size? Or could there be a typo in the Maximum Network Buffer Size setting on one side? Thanks for all, I will find out a solution Best regards PLASSE Lionel | Networks & Systems Administrator 221 Allée de Fétan 01600 TREVOUX - FRANCE Tel : +33(0)4.37.49.91.39 pla...@cofiem.fr www.cofiem.fr | www.cofiem-robotics.fr -Message d'origine- De : Josh Fisher via Bacula-users Envoyé : mardi 7 novembre 2023 18:01 À : bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] fd lose connection On 11/7/23 04:34, Lionel PLASSE wrote: Hello , Could Encryption have any impact in my problem. I am testing without any encryption between SD/DIR/BConsole or FD and it seems to be more stable. (short sized job right done , longest job already running : 750Gb to migrate) My WAN connection seems to be quite good, I achieve transferring big and small raw files by scp ssh and don't have ping latency or troubles with the ipsec connection. So it is fine when the NIC is up. Since this is Windows, the first thing to do is turn off power saving for the network interface device in Device Manager. Make sure that the NIC doesn't ever power down its PHY. If any switch, router, or VPN doesn't handle energy-efficient internet in the same way, then it can look like a dropped connection to the other side. Also, you don't say what type of WAN connection this is. Many wireless services, 5G, etc. can and will drop open sockets due to inactivity (or perceived inactivity) to free up channels. I tried too with NAT, by not using IPSEC and setting Bacula SD & DIR directly in front of the WAN. And the same occurs (wrote X byte but only Y accepted) I Tried too to make a migration job to migrate from SD to SD through WAN instead of SD-> FD through WAN and the result was the same. (to see if win32 FD could be involved) - DIR and SD in the same LAN. - Backup remote FD through remote SD, the two are in the same LAN to fast backup : step OK . - Then migration from remote SD to the SD that is in the DIR area through WAN to outsource volumes physical support : step nok The final goal: outsourcing volumes. I then discard the gzip compression (just in case) The errors are quite disturbing : * Error: lib/bsock.c:397 Wrote 65566 bytes to client:192.168.0.4:9102, but only 65536 accepted Fatal error: filed/backup.c:1008 Network send error to SD. ERR=Input/output error Or (when increasing MaximumNetworkBuffer) * Error: lib/bsock.c:397 Wrote 130277 bytes to client:192.168.0.17:9102, but only 98304 accepted. Fatal error: filed/backup.c:1008 Network send error to SD. ERR=Input/output error Or (Migration job) * Fatal error: append.c:319 Network error reading from FD. ERR=Erreur d'entrée/sortie Error: bsock.c:571 Read expected 131118 got 114684 from Storage daemon:192.168.10.54:9103 It's look like there is a gap between send and receive buffer and looking at the source code, encryption could affect buffer size due to encryption. So I think Bacula-SD could be in cause (maybe). Could it be a bug? What could I do to determine the problem (activating debug in sd daemon ? ) I use Bacula 13.0.3 on Debian 12 , with ssl 1.1 Thank for helping. Backup scenarios must have an a step of relocating the backup media to be reliable. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mail
Re: [Bacula-users] fd lose connection
I’m sorry but no wi-fi nor 5G in our factory, and don't use my phone too to backup my servers :) . I was talking about ssh (scp) transfer just to Just to show out I have no problem when uploading big continuous data using other tools through this wan. The wan connection is quite stable. "So it is fine when the NIC is up. Since this is Windows," no windows. I discarder windows problem hypothesis by using a migration job, so from linux sd to linux sd Thanks for all, I will find out a solution Best regards PLASSE Lionel | Networks & Systems Administrator 221 Allée de Fétan 01600 TREVOUX - FRANCE Tel : +33(0)4.37.49.91.39 pla...@cofiem.fr www.cofiem.fr | www.cofiem-robotics.fr -Message d'origine- De : Josh Fisher via Bacula-users Envoyé : mardi 7 novembre 2023 18:01 À : bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] fd lose connection On 11/7/23 04:34, Lionel PLASSE wrote: > Hello , > > Could Encryption have any impact in my problem. > > I am testing without any encryption between SD/DIR/BConsole or FD and > it seems to be more stable. (short sized job right done , longest job > already running : 750Gb to migrate) > > My WAN connection seems to be quite good, I achieve transferring big and > small raw files by scp ssh and don't have ping latency or troubles with the > ipsec connection. So it is fine when the NIC is up. Since this is Windows, the first thing to do is turn off power saving for the network interface device in Device Manager. Make sure that the NIC doesn't ever power down its PHY. If any switch, router, or VPN doesn't handle energy-efficient internet in the same way, then it can look like a dropped connection to the other side. Also, you don't say what type of WAN connection this is. Many wireless services, 5G, etc. can and will drop open sockets due to inactivity (or perceived inactivity) to free up channels. > > I tried too with NAT, by not using IPSEC and setting Bacula SD & DIR > directly in front of the WAN. > And the same occurs (wrote X byte but only Y accepted) > > I Tried too to make a migration job to migrate from SD to SD through WAN > instead of SD-> FD through WAN and the result was the same. (to see if win32 > FD could be involved) > - DIR and SD in the same LAN. > - Backup remote FD through remote SD, the two are in the same LAN to fast > backup : step OK . > - Then migration from remote SD to the SD that is in the DIR area > through WAN to outsource volumes physical support : step nok The final goal: > outsourcing volumes. > I then discard the gzip compression (just in case) > > The errors are quite disturbing : > * Error: lib/bsock.c:397 Wrote 65566 bytes to client:192.168.0.4:9102, > but only 65536 accepted > Fatal error: filed/backup.c:1008 Network send error to SD. > ERR=Input/output error Or (when increasing MaximumNetworkBuffer) > * Error: lib/bsock.c:397 Wrote 130277 bytes to > client:192.168.0.17:9102, but only 98304 accepted. > Fatal error: filed/backup.c:1008 Network send error to SD. > ERR=Input/output error Or (Migration job) > * Fatal error: append.c:319 Network error reading from FD. ERR=Erreur > d'entrée/sortie > Error: bsock.c:571 Read expected 131118 got 114684 from > Storage daemon:192.168.10.54:9103 > > It's look like there is a gap between send and receive buffer and looking at > the source code, encryption could affect buffer size due to encryption. > So I think Bacula-SD could be in cause (maybe). > Could it be a bug? > What could I do to determine the problem (activating debug in sd > daemon ? ) > > I use Bacula 13.0.3 on Debian 12 , with ssl 1.1 > > Thank for helping. Backup scenarios must have an a step of relocating the > backup media to be reliable. > > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] fd lose connection
On 11/7/23 04:34, Lionel PLASSE wrote: Hello , Could Encryption have any impact in my problem. I am testing without any encryption between SD/DIR/BConsole or FD and it seems to be more stable. (short sized job right done , longest job already running : 750Gb to migrate) My WAN connection seems to be quite good, I achieve transferring big and small raw files by scp ssh and don't have ping latency or troubles with the ipsec connection. So it is fine when the NIC is up. Since this is Windows, the first thing to do is turn off power saving for the network interface device in Device Manager. Make sure that the NIC doesn't ever power down its PHY. If any switch, router, or VPN doesn't handle energy-efficient internet in the same way, then it can look like a dropped connection to the other side. Also, you don't say what type of WAN connection this is. Many wireless services, 5G, etc. can and will drop open sockets due to inactivity (or perceived inactivity) to free up channels. I tried too with NAT, by not using IPSEC and setting Bacula SD & DIR directly in front of the WAN. And the same occurs (wrote X byte but only Y accepted) I Tried too to make a migration job to migrate from SD to SD through WAN instead of SD-> FD through WAN and the result was the same. (to see if win32 FD could be involved) - DIR and SD in the same LAN. - Backup remote FD through remote SD, the two are in the same LAN to fast backup : step OK . - Then migration from remote SD to the SD that is in the DIR area through WAN to outsource volumes physical support : step nok The final goal: outsourcing volumes. I then discard the gzip compression (just in case) The errors are quite disturbing : * Error: lib/bsock.c:397 Wrote 65566 bytes to client:192.168.0.4:9102, but only 65536 accepted Fatal error: filed/backup.c:1008 Network send error to SD. ERR=Input/output error Or (when increasing MaximumNetworkBuffer) * Error: lib/bsock.c:397 Wrote 130277 bytes to client:192.168.0.17:9102, but only 98304 accepted. Fatal error: filed/backup.c:1008 Network send error to SD. ERR=Input/output error Or (Migration job) * Fatal error: append.c:319 Network error reading from FD. ERR=Erreur d'entrée/sortie Error: bsock.c:571 Read expected 131118 got 114684 from Storage daemon:192.168.10.54:9103 It's look like there is a gap between send and receive buffer and looking at the source code, encryption could affect buffer size due to encryption. So I think Bacula-SD could be in cause (maybe). Could it be a bug? What could I do to determine the problem (activating debug in sd daemon ? ) I use Bacula 13.0.3 on Debian 12 , with ssl 1.1 Thank for helping. Backup scenarios must have an a step of relocating the backup media to be reliable. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] fd lose connection
Hello , Could Encryption have any impact in my problem. I am testing without any encryption between SD/DIR/BConsole or FD and it seems to be more stable. (short sized job right done , longest job already running : 750Gb to migrate) My WAN connection seems to be quite good, I achieve transferring big and small raw files by scp ssh and don't have ping latency or troubles with the ipsec connection. I tried too with NAT, by not using IPSEC and setting Bacula SD & DIR directly in front of the WAN. And the same occurs (wrote X byte but only Y accepted) I Tried too to make a migration job to migrate from SD to SD through WAN instead of SD-> FD through WAN and the result was the same. (to see if win32 FD could be involved) - DIR and SD in the same LAN. - Backup remote FD through remote SD, the two are in the same LAN to fast backup : step OK . - Then migration from remote SD to the SD that is in the DIR area through WAN to outsource volumes physical support : step nok The final goal: outsourcing volumes. I then discard the gzip compression (just in case) The errors are quite disturbing : * Error: lib/bsock.c:397 Wrote 65566 bytes to client:192.168.0.4:9102, but only 65536 accepted Fatal error: filed/backup.c:1008 Network send error to SD. ERR=Input/output error Or (when increasing MaximumNetworkBuffer) * Error: lib/bsock.c:397 Wrote 130277 bytes to client:192.168.0.17:9102, but only 98304 accepted. Fatal error: filed/backup.c:1008 Network send error to SD. ERR=Input/output error Or (Migration job) * Fatal error: append.c:319 Network error reading from FD. ERR=Erreur d'entrée/sortie Error: bsock.c:571 Read expected 131118 got 114684 from Storage daemon:192.168.10.54:9103 It's look like there is a gap between send and receive buffer and looking at the source code, encryption could affect buffer size due to encryption. So I think Bacula-SD could be in cause (maybe). Could it be a bug? What could I do to determine the problem (activating debug in sd daemon ? ) I use Bacula 13.0.3 on Debian 12 , with ssl 1.1 Thank for helping. Backup scenarios must have an a step of relocating the backup media to be reliable. ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] fd lose connection
Job lost connection ... -Message d'origine- De : Lionel PLASSE Envoyé : dimanche 5 novembre 2023 08:00 À : Lionel PLASSE ; Bill Arlofski ; bacula-users@lists.sourceforge.net Objet : RE: [Bacula-users] fd lose connection I upgraded net device drivers to last version I change some network adapters conf (Broadcom netxtrem2) and gain some Mb/s (up to 10 now) Backup still running. Will see if it achieve job well -Message d'origine- De : Lionel PLASSE Envoyé : vendredi 3 novembre 2023 22:02 À : Bill Arlofski ; bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] fd lose connection Yes , you're right But the job work well for an other machine through the same vpn and the job last 9hours . And from the hardware vpn gateway, I don't have much configuration choice. It could be a direction. Could it be a problem of Debian 12 which seems not to be official for Bacula v13 ? "SD calls Client" directive was activated , don't know why ( I tried so many conf to solve it :). I will re-test without And for the slow rate I am trying the Network buffer size directive on FD and SD but not really conclusive. But : Error: lib/bsock.c:397 Wrote 65566 bytes to client:192.168.0.4:9102, but only 65536 accepted Tells me to look after the send and receive buffer on network drivers Bacula doc : Note, on certain Windows machines, there are reports that the transfer rates are very slow and this seems to be related to the default 65,536 size. On systems where the transfer rates seem abnormally slow compared to other systems, you might try setting the Maximum Network Buffer Size to 32,768 in both the File daemon and in the Storage daemon ... The default size was chosen to be relatively large but not too big in the case that you are transmitting data over Internet. It is clear that on a high speed local network, you can increase this number and improve performance. For example, some users have found that if you use a value of 65,536 bytes they get five to ten times the throughput. Larger values for most users don’t seem to improve performance. If you are interested in improving your backup speeds, this is definitely a place to experiment. You will probably also want to make the corresponding change in each of your File daemons conf files. I'm disappointed, and made so many inoperant conf changes , that I will try to with v9.6.6 ( I used before on my lan) to see if the problem is the same . ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] fd lose connection
I upgraded net device drivers to last version I change some network adapters conf (Broadcom netxtrem2) and gain some Mb/s (up to 10 now) Backup still running. Will see if it achieve job well -Message d'origine- De : Lionel PLASSE Envoyé : vendredi 3 novembre 2023 22:02 À : Bill Arlofski ; bacula-users@lists.sourceforge.net Objet : Re: [Bacula-users] fd lose connection Yes , you're right But the job work well for an other machine through the same vpn and the job last 9hours . And from the hardware vpn gateway, I don't have much configuration choice. It could be a direction. Could it be a problem of Debian 12 which seems not to be official for Bacula v13 ? "SD calls Client" directive was activated , don't know why ( I tried so many conf to solve it :). I will re-test without And for the slow rate I am trying the Network buffer size directive on FD and SD but not really conclusive. But : Error: lib/bsock.c:397 Wrote 65566 bytes to client:192.168.0.4:9102, but only 65536 accepted Tells me to look after the send and receive buffer on network drivers Bacula doc : Note, on certain Windows machines, there are reports that the transfer rates are very slow and this seems to be related to the default 65,536 size. On systems where the transfer rates seem abnormally slow compared to other systems, you might try setting the Maximum Network Buffer Size to 32,768 in both the File daemon and in the Storage daemon ... The default size was chosen to be relatively large but not too big in the case that you are transmitting data over Internet. It is clear that on a high speed local network, you can increase this number and improve performance. For example, some users have found that if you use a value of 65,536 bytes they get five to ten times the throughput. Larger values for most users don’t seem to improve performance. If you are interested in improving your backup speeds, this is definitely a place to experiment. You will probably also want to make the corresponding change in each of your File daemons conf files. I'm disappointed, and made so many inoperant conf changes , that I will try to with v9.6.6 ( I used before on my lan) to see if the problem is the same . smime.p7s Description: S/MIME cryptographic signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] fd lose connection
Yes , you're right But the job work well for an other machine through the same vpn and the job last 9hours . And from the hardware vpn gateway, I don't have much configuration choice. It could be a direction. Could it be a problem of Debian 12 which seems not to be official for Bacula v13 ? "SD calls Client" directive was activated , don't know why ( I tried so many conf to solve it :). I will re-test without And for the slow rate I am trying the Network buffer size directive on FD and SD but not really conclusive. But : Error: lib/bsock.c:397 Wrote 65566 bytes to client:192.168.0.4:9102, but only 65536 accepted Tells me to look after the send and receive buffer on network drivers Bacula doc : Note, on certain Windows machines, there are reports that the transfer rates are very slow and this seems to be related to the default 65,536 size. On systems where the transfer rates seem abnormally slow compared to other systems, you might try setting the Maximum Network Buffer Size to 32,768 in both the File daemon and in the Storage daemon ... The default size was chosen to be relatively large but not too big in the case that you are transmitting data over Internet. It is clear that on a high speed local network, you can increase this number and improve performance. For example, some users have found that if you use a value of 65,536 bytes they get five to ten times the throughput. Larger values for most users don’t seem to improve performance. If you are interested in improving your backup speeds, this is definitely a place to experiment. You will probably also want to make the corresponding change in each of your File daemons conf files. I'm disappointed, and made so many inoperant conf changes , that I will try to with v9.6.6 ( I used before on my lan) to see if the problem is the same . smime.p7s Description: S/MIME cryptographic signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] fd lose connection
Lionel, I found this post from 10 years ago. That user had a similar problem. Kern suggested add "Heartbeat Interval = 300" to the appropriate resources on both sides (dir and FD in your case). I see you mentioned setting heartbeat, did you set it on both sides? Source: old post to this list https://bacula-users.narkive.com/i6Y7gQ1J/lib-bsock-c-397-write-error-sending-1-bytes-to-storage-daemon Robert Gerber 402-237-8692 r...@craeon.net On Fri, Nov 3, 2023, 1:51 PM Lionel PLASSE wrote: > Hello , > > I am losing connection between sd and windows fd and I don't understand > why. > The machine lose every time the connection, sometimes after an hour and > often after 2 hours of job. > And the transfer rate is so slow around 3 to 5 Mb/s (instead of up to 50 in > other configurations) > > I tried many Network Maximum Rate values (32768 and higher and smaller) but > every time I have more byte written to client (65566 > 65536) and > connection > fall down. > > The server is in win2019 version with bacula-fd win32 v13 and the Bacula > Dir is in v13.0.3 on a Debian 12 (with libssl1.1 and 6.1.0-13-amd kernel) > > I before was using bacula v9.6 and all wad good through my Lan network > when > backing up this same windows server. > > I am now using a Debian server in a different place, through a site to > site > permanent ipsec VPN ( general upload wan bandwith 80Mb/s) > The jobs are good for the other machines (win2012-r2 & Debian 10) which > reach a rate of 10 up to 30 Mb/s depending the time and size of files > backed > up. > > But for this one (and another one in win2008-r2) : > serv-dc-01-fd JobId 122: Fatal error: filed/backup.c:1008 Network send > error > to SD. ERR=Input/output error > serv-dc-01-fd JobId 122: Error: lib/bsock.c:397 Wrote 65566 bytes to > client:192.168.0.4:9102, but only 65536 accepted > > I check full duplex, opened firewall port, tcp_keep_alive & heartbeat, > changed net buffer size,installed last driver . > > Any suggestion ? > ___ > Bacula-users mailing list > Bacula-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bacula-users > ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] fd lose connection
On 11/3/23 13:17, Rob Gerber wrote: Oh, reading the error message again, did you set "heartbeat interval = 300" between Dir and SD? Your error message is from the SD. I don't know if FD will communicate directly with SD or if data from Fd goes through dir to SD. Robert Gerber Hello Robert, Just an FYI, a HeartbeatInterval = 300 seconds has been hard-coded as the default - even when not explicitly set - for several years now. :) He may want to set a lower one on each Daemon (DIR, SD, FD), but I know from testing that anything less than 60 will be ignored and 60 seconds will be used. ;) But it really sounds to me that Lionel's problem is possibly an overly aggressive firewall closing sockets or it could also have something to do with the IPsec VPN - since that seems to be the only real netowrking change - so that is where I would look first. Best regards, Bill -- Bill Arlofski w...@protonmail.com signature.asc Description: OpenPGP digital signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
Re: [Bacula-users] fd lose connection
Oh, reading the error message again, did you set "heartbeat interval = 300" between Dir and SD? Your error message is from the SD. I don't know if FD will communicate directly with SD or if data from Fd goes through dir to SD. Robert Gerber 402-237-8692 r...@craeon.net On Fri, Nov 3, 2023, 2:15 PM Rob Gerber wrote: > Lionel, > > I found this post from 10 years ago. That user had a similar problem. Kern > suggested add "Heartbeat Interval = 300" to the appropriate resources on > both sides (dir and FD in your case). I see you mentioned setting > heartbeat, did you set it on both sides? > > Source: old post to this list > > https://bacula-users.narkive.com/i6Y7gQ1J/lib-bsock-c-397-write-error-sending-1-bytes-to-storage-daemon > > > Robert Gerber > 402-237-8692 > r...@craeon.net > > On Fri, Nov 3, 2023, 1:51 PM Lionel PLASSE wrote: > >> Hello , >> >> I am losing connection between sd and windows fd and I don't understand >> why. >> The machine lose every time the connection, sometimes after an hour and >> often after 2 hours of job. >> And the transfer rate is so slow around 3 to 5 Mb/s (instead of up to 50 >> in >> other configurations) >> >> I tried many Network Maximum Rate values (32768 and higher and smaller) >> but >> every time I have more byte written to client (65566 > 65536) and >> connection >> fall down. >> >> The server is in win2019 version with bacula-fd win32 v13 and the Bacula >> Dir is in v13.0.3 on a Debian 12 (with libssl1.1 and 6.1.0-13-amd kernel) >> >> I before was using bacula v9.6 and all wad good through my Lan network >> when >> backing up this same windows server. >> >> I am now using a Debian server in a different place, through a site to >> site >> permanent ipsec VPN ( general upload wan bandwith 80Mb/s) >> The jobs are good for the other machines (win2012-r2 & Debian 10) which >> reach a rate of 10 up to 30 Mb/s depending the time and size of files >> backed >> up. >> >> But for this one (and another one in win2008-r2) : >> serv-dc-01-fd JobId 122: Fatal error: filed/backup.c:1008 Network send >> error >> to SD. ERR=Input/output error >> serv-dc-01-fd JobId 122: Error: lib/bsock.c:397 Wrote 65566 bytes to >> client:192.168.0.4:9102, but only 65536 accepted >> >> I check full duplex, opened firewall port, tcp_keep_alive & heartbeat, >> changed net buffer size,installed last driver . >> >> Any suggestion ? >> ___ >> Bacula-users mailing list >> Bacula-users@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bacula-users >> > ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
[Bacula-users] fd lose connection
Hello , I am losing connection between sd and windows fd and I don't understand why. The machine lose every time the connection, sometimes after an hour and often after 2 hours of job. And the transfer rate is so slow around 3 to 5 Mb/s (instead of up to 50 in other configurations) I tried many Network Maximum Rate values (32768 and higher and smaller) but every time I have more byte written to client (65566 > 65536) and connection fall down. The server is in win2019 version with bacula-fd win32 v13 and the Bacula Dir is in v13.0.3 on a Debian 12 (with libssl1.1 and 6.1.0-13-amd kernel) I before was using bacula v9.6 and all wad good through my Lan network when backing up this same windows server. I am now using a Debian server in a different place, through a site to site permanent ipsec VPN ( general upload wan bandwith 80Mb/s) The jobs are good for the other machines (win2012-r2 & Debian 10) which reach a rate of 10 up to 30 Mb/s depending the time and size of files backed up. But for this one (and another one in win2008-r2) : serv-dc-01-fd JobId 122: Fatal error: filed/backup.c:1008 Network send error to SD. ERR=Input/output error serv-dc-01-fd JobId 122: Error: lib/bsock.c:397 Wrote 65566 bytes to client:192.168.0.4:9102, but only 65536 accepted I check full duplex, opened firewall port, tcp_keep_alive & heartbeat, changed net buffer size,installed last driver . Any suggestion ? smime.p7s Description: S/MIME cryptographic signature ___ Bacula-users mailing list Bacula-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/bacula-users
