It may well be just that I can't see the wood for the trees when looking at logging but I can't find the problem :-(
I'm running daily manual dumps of the FAILED DLE's to keep backups intact!
I'm still getting the following:
FAILURE DUMP SUMMARY:
bentley Resources lev 1 FAILED [too many dumper retry: [request failed: No
route to host]]
bentley sysadmin lev 1 FAILED [too many dumper retry: [request failed: No
route to host]]
Apart from the two KVM hosts, all these systems are KVM Guests. The backup
server is a KVM guest.
Has anyone seen or know of issues that may occur with amanda on virtualised
infrastructure?
From my understanding of KVM networking between guests, whole network frames
are dumped and picked
up between them. This allows higher transport speeds. I've tested the
throughput with iperf and have
seen througput as high as 25Gbps. The following ipef session shows the
connection between the failed
guest, bentley, and the backup server. I've only shown the 'server' side
results for iperf below:
# systemctl stop xinetd
# iperf -p 10080 -s
------------------------------------------------------------
Server listening on TCP port 10080
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 4] local 10.0.19.21 port 10080 connected with 192.168.0.3 port 39214
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 20.5 GBytes 17.6 Gbits/sec
[ 4] local 10.0.19.21 port 10080 connected with 192.168.0.3 port 39215
[ 4] 0.0-10.0 sec 20.7 GBytes 17.8 Gbits/sec
[ 4] local 10.0.19.21 port 10080 connected with 192.168.0.3 port 39218
[ 4] 0.0-10.0 sec 21.3 GBytes 18.3 Gbits/sec
[ 4] local 10.0.19.21 port 10080 connected with 192.168.0.3 port 39223
[ 4] 0.0-10.0 sec 21.4 GBytes 18.4 Gbits/sec
Any clues/help for the above are appreciated.
I'm now also getting some other strange errors that I've never seen before.
These report as 'FAILED'
but further on into the report they appear to have completed without issue.
What do the error codes
signify (e.g. FAILED [02-00098] etc.)?
---8<---
FAILURE DUMP SUMMARY:
---8<---
bentley ECN lev 0 FAILED [02-00098]
bentley Repair lev 1 FAILED [06-00229]
garage /var lev 1 FAILED [shm_ring cancelled]
modena /usr/src lev 1 FAILED [12-00205]
---8<---
NOTES:
planner: Last full dump of bentley:ECN on tape daily02 overwritten in 5 runs.
planner: Last level 1 dump of bentley:ECN on tape daily01 overwritten in 4
runs.
planner: Last full dump of bentley:Repair on tape daily07 overwritten in 2
runs.
planner: Last full dump of garage:/var on tape daily01 overwritten in 4 runs.
---8<---
DUMP SUMMARY:
DUMPER STATS
TAPER STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS
KB/s MMM:SS KB/s
------------------------------------ ---------------------------
--------------- ---------------
---8<---
bentley ECN 0 19790 19790 -- 0:03
7325.0 0:00 197900.0
bentley Repair 1 10 0.0 0:00
4.2 0:00 0.0
garage /var 1 7000 7000 -- 0:00
33341.0 0:00 70000.0
modena /usr/src 1 190 14 7.4 0:04
3.3 0:00 140.0
---8<---
What are the error codes and did amanda dump these OK or not?
Kind regards,
Tom
Tom Robinson
IT Manager/System Administrator
MoTeC Pty Ltd
121 Merrindale Drive
Croydon South
3136 Victoria
Australia
T: +61 3 9761 5050
F: +61 3 9761 5051
E: [email protected]
On 13/09/17 23:09, Jean-Louis Martineau wrote:
> Tom,
>
> It is the system that return the "No route to host" error.
> You should check your system log (on server, client, router, firewall, nat,
> ...) for network error.
>
> Jean-Louis
>
> On 12/09/17 06:01 PM, Tom Robinson wrote:
>> bump
>>
>> On 11/09/17 12:45, Tom Robinson wrote:
>> > Hi,
>> >
>> > I've recently migrated our backup server from CentOS 5 to CentOS 7. I've
>> > also upgraded from amanda
>> > 3.3.7 to 3.4.5
>> >
>> > The amcheck works fine and reports no issues. Yet, on backup runs on some
>> > DLEs I get the error:
>> >
>> > dump failed: [request failed: No route to host](too)
>> >
>> > It also appears to be random as to which DLEs fail. Sometimes it's just
>> > one or two on a client.
>> > Other times it's all DLEs for a client. And, for any particular client it
>> > can be a different DLE on
>> > that client each day.
>> >
>> > Below is a dumper..debug log from the server. I'm not sure what to check
>> > for in there. What other
>> > logs should I check?
>> >
>> > Kind regards,
>> > Tom
>> >
>> > Sun Sep 10 20:16:32.115899592 2017: pid 6088: thd-0x257f400: dumper:
>> > close_producer_shm_ring
>> > sem_close(sem_write 0x7fbc1588b000
>> > Sun Sep 10 20:16:32.115911222 2017: pid 6088: thd-0x257f400: dumper:
>> > am_sem_close 0x7fbc1588b000 0
>> > Sun Sep 10 20:16:32.115927349 2017: pid 6088: thd-0x257f400: dumper:
>> > am_sem_close 0x7fbc15889000 0
>> > Sun Sep 10 20:16:32.115938800 2017: pid 6088: thd-0x257f400: dumper:
>> > am_sem_close 0x7fbc1588a000 0
>> > Sun Sep 10 20:16:32.115949293 2017: pid 6088: thd-0x257f400: dumper:
>> > am_sem_close 0x7fbc15888000 0
>> > Sun Sep 10 20:16:32.337361676 2017: pid 6088: thd-0x257f400: dumper:
>> > getcmd: SHM-DUMP 00-00217
>> 34076
>> > NULL 5 bentley ffffffff9efefbffffffffff3f000000000000 Dispatch
>> /var/lib/samba/data/public/Dispatch 1
>> > 2017:9:6:4:6:22 GNUTAR "" "" "" "" "" "" "" 1 "" "" bsdtcp AMANDA /amand
>> > a_shm_control-6956-0 20 |" <auth>bsdtcp</auth>\n
>> > <compress>FAST</compress>\n
>> > <record>YES</record>\n <index>YES</index>\n
>> > <datapath>AMANDA</datapath>\n"""
>> > Sun Sep 10 20:16:32.337507787 2017: pid 6088: thd-0x257f400: dumper:
>> > Sending header to
>> localhost:34076
>> > Sun Sep 10 20:16:32.339939372 2017: pid 6088: thd-0x257f400: dumper:
>> > make_socket opening socket
>> with
>> > family 10
>> > Sun Sep 10 20:16:32.339978452 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_port: Try port 1024:
>> > available - Success
>> > Sun Sep 10 20:16:32.340075462 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: Connect
>> from
>> > :::1024 failed: Connection refused
>> > Sun Sep 10 20:16:32.340101209 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: connect to
>> > ::1:34076 failed: Connection refused
>> > Sun Sep 10 20:16:32.342383119 2017: pid 6088: thd-0x257f400: dumper:
>> > make_socket opening socket
>> with
>> > family 2
>> > Sun Sep 10 20:16:32.342418634 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_port: Try port 1024:
>> > available - Success
>> > Sun Sep 10 20:16:32.342489613 2017: pid 6088: thd-0x257f400: dumper:
>> > connected to
>> 127.0.0.1:34076 <http://127.0.0.1:34076>
>> > Sun Sep 10 20:16:32.342501059 2017: pid 6088: thd-0x257f400: dumper: our
>> > side is 0.0.0.0:1024
>> <http://0.0.0.0:1024>
>> > Sun Sep 10 20:16:32.342509347 2017: pid 6088: thd-0x257f400: dumper:
>> > try_socksize: send buffer size
>> > is 131072
>> > Sun Sep 10 20:16:32.342558663 2017: pid 6088: thd-0x257f400: dumper: send
>> > request:
>> > ----
>> > SERVICE sendbackup
>> > OPTIONS
>> > features=ffffffff9efefbfffffffffffffff3fffbf70f;maxdumps=5;hostname=bentley;config=daily;
>> > <dle>
>> > <program>GNUTAR</program>
>> > <disk>Dispatch</disk>
>> > <diskdevice>/var/lib/samba/data/public/Dispatch</diskdevice>
>> > <level>1</level>
>> > <auth>bsdtcp</auth>
>> > <compress>FAST</compress>
>> > <record>YES</record>
>> > <index>YES</index>
>> > <datapath>AMANDA</datapath>
>> > </dle>
>> >
>> > ----
>> >
>> > Sun Sep 10 20:16:32.342572947 2017: pid 6088: thd-0x257f400: dumper:
>> security_getdriver(name=bsdtcp)
>> > returns 0x7fbc153e86a0
>> > Sun Sep 10 20:16:32.342582472 2017: pid 6088: thd-0x257f400: dumper:
>> > security_handleinit(handle=0x25e2e70, driver=0x7fbc153e86a0 (BSDTCP))
>> > Sun Sep 10 20:16:32.343623490 2017: pid 6088: thd-0x257f400: dumper:
>> > security_streaminit(stream=0x283d6e0, driver=0x7fbc153e86a0 (BSDTCP))
>> > Sun Sep 10 20:16:32.346176806 2017: pid 6088: thd-0x257f400: dumper:
>> > make_socket opening socket
>> with
>> > family 2
>> > Sun Sep 10 20:16:32.346230063 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_port: Try port 571:
>> > available - Success
>> > Sun Sep 10 20:16:32.346247716 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: Connect
>> from
>> > 0.0.0.0:571 <http://0.0.0.0:571> failed: Cannot assign requested address
>> > Sun Sep 10 20:16:32.346261235 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: connect to
>> > 192.168.0.3:10080 <http://192.168.0.3:10080> failed: Cannot assign
>> > requested address
>> > Sun Sep 10 20:16:32.348492651 2017: pid 6088: thd-0x257f400: dumper:
>> > make_socket opening socket
>> with
>> > family 2
>> > Sun Sep 10 20:16:32.348526207 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_port: Try port 585:
>> > available - Success
>> > Sun Sep 10 20:18:39.587177652 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: Connect
>> from
>> > 0.0.0.0:585 <http://0.0.0.0:585> failed: Connection timed out
>> > Sun Sep 10 20:18:39.587235409 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: connect to
>> > 192.168.0.3:10080 <http://192.168.0.3:10080> failed: Connection timed out
>> > Sun Sep 10 20:18:39.587267623 2017: pid 6088: thd-0x257f400: dumper:
>> > stream_client: Could not bind
>> > to port in range 512-1023.
>> > Sun Sep 10 20:18:39.587290672 2017: pid 6088: thd-0x257f400: dumper:
>> > security_seterror(handle=0x25e2e70, driver=0x7fbc153e86a0 (BSDTCP)
>> > error=Connection timed out)
>> > Sun Sep 10 20:18:39.587299769 2017: pid 6088: thd-0x257f400: dumper:
>> > security_close(handle=0x25e2e70, driver=0x7fbc153e86a0 (BSDTCP))
>> > Sun Sep 10 20:18:39.587308804 2017: pid 6088: thd-0x257f400: dumper:
>> security_stream_close(0x283d6e0)
>> > Sun Sep 10 20:18:44.592385871 2017: pid 6088: thd-0x257f400: dumper:
>> > security_handleinit(handle=0x25def50, driver=0x7fbc153e86a0 (BSDTCP))
>> > Sun Sep 10 20:18:44.593300425 2017: pid 6088: thd-0x257f400: dumper:
>> > security_streaminit(stream=0x2845d10, driver=0x7fbc153e86a0 (BSDTCP))
>> > Sun Sep 10 20:18:44.596042201 2017: pid 6088: thd-0x257f400: dumper:
>> > make_socket opening socket
>> with
>> > family 2
>> > Sun Sep 10 20:18:44.596123530 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_port: Try port 571:
>> > available - Success
>> > Sun Sep 10 20:18:44.596155627 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: Connect
>> from
>> > 0.0.0.0:571 <http://0.0.0.0:571> failed: Cannot assign requested address
>> > Sun Sep 10 20:18:44.596163545 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: connect to
>> > 192.168.0.3:10080 <http://192.168.0.3:10080> failed: Cannot assign
>> > requested address
>> > Sun Sep 10 20:18:44.598648813 2017: pid 6088: thd-0x257f400: dumper:
>> > make_socket opening socket
>> with
>> > family 2
>> > Sun Sep 10 20:18:44.598686408 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_port: Try port 585:
>> > available - Success
>> > Sun Sep 10 20:18:44.598811157 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: Connect
>> from
>> > 0.0.0.0:585 <http://0.0.0.0:585> failed: No route to host
>> > Sun Sep 10 20:18:44.598823545 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: connect to
>> > 192.168.0.3:10080 <http://192.168.0.3:10080> failed: No route to host
>> > Sun Sep 10 20:18:44.598852583 2017: pid 6088: thd-0x257f400: dumper:
>> > stream_client: Could not bind
>> > to port in range 512-1023.
>> > Sun Sep 10 20:18:44.598866659 2017: pid 6088: thd-0x257f400: dumper:
>> > security_seterror(handle=0x25def50, driver=0x7fbc153e86a0 (BSDTCP)
>> > error=No route to host)
>> > Sun Sep 10 20:18:44.598873575 2017: pid 6088: thd-0x257f400: dumper:
>> > security_close(handle=0x25def50, driver=0x7fbc153e86a0 (BSDTCP))
>> > Sun Sep 10 20:18:44.598880166 2017: pid 6088: thd-0x257f400: dumper:
>> security_stream_close(0x2845d10)
>> > Sun Sep 10 20:18:49.603142410 2017: pid 6088: thd-0x257f400: dumper:
>> > security_handleinit(handle=0x25def50, driver=0x7fbc153e86a0 (BSDTCP))
>> > Sun Sep 10 20:18:49.604135032 2017: pid 6088: thd-0x257f400: dumper:
>> > security_streaminit(stream=0x284e340, driver=0x7fbc153e86a0 (BSDTCP))
>> > Sun Sep 10 20:18:49.606574107 2017: pid 6088: thd-0x257f400: dumper:
>> > make_socket opening socket
>> with
>> > family 2
>> > Sun Sep 10 20:18:49.606626709 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_port: Try port 571:
>> > available - Success
>> > Sun Sep 10 20:18:49.606645124 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: Connect
>> from
>> > 0.0.0.0:571 <http://0.0.0.0:571> failed: Cannot assign requested address
>> > Sun Sep 10 20:18:49.606652941 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: connect to
>> > 192.168.0.3:10080 <http://192.168.0.3:10080> failed: Cannot assign
>> > requested address
>> > Sun Sep 10 20:18:49.609070482 2017: pid 6088: thd-0x257f400: dumper:
>> > make_socket opening socket
>> with
>> > family 2
>> > Sun Sep 10 20:18:49.609113309 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_port: Try port 585:
>> > available - Success
>> > Sun Sep 10 20:18:49.609232279 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: Connect
>> from
>> > 0.0.0.0:585 <http://0.0.0.0:585> failed: No route to host
>> > Sun Sep 10 20:18:49.609244774 2017: pid 6088: thd-0x257f400: dumper:
>> > connect_portrange: connect to
>> > 192.168.0.3:10080 <http://192.168.0.3:10080> failed: No route to host
>> > Sun Sep 10 20:18:49.609256366 2017: pid 6088: thd-0x257f400: dumper:
>> > stream_client: Could not bind
>> > to port in range 512-1023.
>> > Sun Sep 10 20:18:49.609267524 2017: pid 6088: thd-0x257f400: dumper:
>> > security_seterror(handle=0x25def50, driver=0x7fbc153e86a0 (BSDTCP)
>> > error=No route to host)
>> > Sun Sep 10 20:18:49.609276028 2017: pid 6088: thd-0x257f400: dumper:
>> > security_close(handle=0x25def50, driver=0x7fbc153e86a0 (BSDTCP))
>> > Sun Sep 10 20:18:49.609282311 2017: pid 6088: thd-0x257f400: dumper:
>> security_stream_close(0x284e340)
>> > Sun Sep 10 20:18:49.609296611 2017: pid 6088: thd-0x257f400: dumper:
>> > putresult: 11 TRY-AGAIN
>> > 00-00217 "[request failed: No route to host]"
>> >
>>
>>
>
>
>
> *Disclaimer*
>
> This message is the property of *CARBONITE, INC.* <http://www.carbonite.com>
> and may contain
> confidential or privileged information.
>
> If this message has been delivered to you by mistake, then do not copy or
> deliver this message to
> anyone. Instead, destroy it and notify me by reply e-mail.
>
signature.asc
Description: OpenPGP digital signature
