Hi Brian,
sure, here is hex output of simultaneously running ping and blackbox
exporter:
sudo tcpdump -i ens160 -n -X icmp and host 8.8.8.8
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
15:51:57.237795 IP 192.168.80.21 > 8.8.8.8: ICMP echo request, id 33313, seq
2136, length 36
0x0000: 4500 0038 ea2e 4000 3f01 30c9 c0a8 5015 E..8..@.?.0...P.
0x0010: 0808 0808 0800 11ec 8221 0858 5072 6f6d .........!.XProm
0x0020: 6574 6865 7573 2042 6c61 636b 626f 7820 etheus.Blackbox.
0x0030: 4578 706f 7274 6572 Exporter
15:51:57.396007 IP 192.168.80.21 > 8.8.8.8: ICMP echo request, id 16978, seq
42826, length 64
0x0000: 4500 0054 2aa5 4000 4001 ef36 c0a8 5015 E..T*.@[email protected].
0x0010: 0808 0808 0800 8da1 4252 a74a 7d85 8c5e ........BR.J}..^
0x0020: 0000 0000 b20a 0600 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
15:51:57.398327 IP 8.8.8.8 > 192.168.80.21: ICMP echo reply, id 16978, seq
42826, length 64
0x0000: 4520 0054 0000 0000 3401 65bc 0808 0808 E..T....4.e.....
0x0010: c0a8 5015 0000 95a1 4252 a74a 7d85 8c5e ..P.....BR.J}..^
0x0020: 0000 0000 b20a 0600 0000 0000 1011 1213 ................
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
To get tcpdump of packets before the ping probe of blacbox exporter failed,
I have to restart the blackbox exporter container and
wait for some time (maybe hours) until the probe will start failing again
(only coincidence at this times are short spikes of DNS outages in
our network)
For now, I have created a droplet on Digital Ocean and add its IP address
as blackbox exporter probe. The probe is now working
ok and will fail after some time as other, I suggest. Here are tcpdumps:
tcpdump on *blackbox exporter machine*:
sudo tcpdump -i ens160 -n -X icmp and host 104.248.242.37
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
16:37:30.997577 IP 192.168.80.21 > 104.248.242.37: ICMP echo request, id 33313,
seq 7358, length 36
0x0000: 4500 0038 7e9d 4000 3f01 514c c0a8 5015 E..8~.@.?.QL..P.
0x0010: 68f8 f225 0800 fd85 8221 1cbe 5072 6f6d h..%.....!..Prom
0x0020: 6574 6865 7573 2042 6c61 636b 626f 7820 etheus.Blackbox.
0x0030: 4578 706f 7274 6572 Exporter
16:37:31.013151 IP 104.248.242.37 > 192.168.80.21: ICMP echo reply, id 33313,
seq 7358, length 36
0x0000: 4520 0038 a8d3 0000 2f01 76f6 68f8 f225 E..8..../.v.h..%
0x0010: c0a8 5015 0000 0586 8221 1cbe 5072 6f6d ..P......!..Prom
0x0020: 6574 6865 7573 2042 6c61 636b 626f 7820 etheus.Blackbox.
0x0030: 4578 706f 7274 6572 Exporter
tcpdump on *remote machine* (DigitalOcean):
root@ubuntu-s-1vcpu-1gb-fra1-01:~# tcpdump -i any -X icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
14:38:51.013616 IP 109.183.26.126 > ubuntu-s-1vcpu-1gb-fra1-01: ICMP echo
request, id 33313, seq 7518, length 36
0x0000: 4500 0038 a44c 4000 3001 c325 6db7 1a7e [email protected]..%m..~
0x0010: 68f8 f225 0800 fce5 8221 1d5e 5072 6f6d h..%.....!.^Prom
0x0020: 6574 6865 7573 2042 6c61 636b 626f 7820 etheus.Blackbox.
0x0030: 4578 706f 7274 6572 Exporter
14:38:51.013648 IP ubuntu-s-1vcpu-1gb-fra1-01 > 109.183.26.126: ICMP echo
reply, id 33313, seq 7518, length 36
0x0000: 4500 0038 d6c0 0000 4001 c0b1 68f8 f225 [email protected]..%
0x0010: 6db7 1a7e 0000 04e6 8221 1d5e 5072 6f6d m..~.....!.^Prom
0x0020: 6574 6865 7573 2042 6c61 636b 626f 7820 etheus.Blackbox.
0x0030: 4578 706f 7274 6572 Exporter
Thank You very much for advice,
I will post remaining output, when the new probe starts to fail ...
Regards,
Tom
Dne úterý 7. dubna 2020 15:24:08 UTC+2 Brian Candler napsal(a):
>
> 15:28:48.734661 IP 172.17.0.5 > 8.8.8.8: ICMP echo request, id 33313, seq
> 41979, length 36
> 0x0000: 4500 0038 f40e 4000 4001 8a90 ac11 0005 E..8..@.@.......
> 0x0010: 0808 0808 0800 7648 8221 a3fb 5072 6f6d ......vH.!..Prom
> 0x0020: 6574 6865 7573 2042 6c61 636b 626f 7820 etheus.Blackbox.
> 0x0030: 4578 706f 7274 6572 Exporter
>
>
> FWIW, I've recalculated the IP and ICMP checksums, and both appear to be
> correct.
>
> You're definitely sure the packets are leaving the machine? Can you try a
> blackbox test of some remote host or VM on the Internet that you control,
> and run tcpdump there to check for incoming ICMP?
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/24b7d922-91e6-4fe8-8c70-9cd6ce6f5727%40googlegroups.com.
