Re: Network tcp/iptables issue with XRDP
Henggi writes: > Oh wow… that’s interesting. I had no idea about „nft“ (I just knew > „iptables-nft“) which seem to be very different. > I think I have dig down where those „nft" rules are coming from while > iptables-nft is completely empty. Thanks, great clue! Typically you'd have a /etc/nftables.conf with the rules for nft. Or at least that's what I do.
Re: Network tcp/iptables issue with XRDP
> On 24 Oct 2023, at 17:36, Arno Lehmann wrote:
>
> Hello,
Hi Arno,
>
> Am 24.10.2023 um 16:19 schrieb Henggi:
> ...
>> As I mentioned in my 1st email, I think (afaik) that no other netfitler
>> module/service is running.
>
> My anecdote was intended to illustrate that beliefs are not really good tools
> to diagnose problems ;-)
fair enough :)
>
>> root@server:~# systemctl status firewalld
>> Unit firewalld.service could not be found.
>
> Ok, so whatever it is, it's not firewalld managed on the local host.
>
> What does
>
> nft list ruleset
>
> show?
> (Unfortunately, this is all I know about netfilter diagnostics... and I
> couldn't even get this far without internet search engine :-)
Oh wow… that’s interesting. I had no idea about „nft“ (I just knew
„iptables-nft“) which seem to be very different.
I think I have dig down where those „nft" rules are coming from while
iptables-nft is completely empty. Thanks, great clue!
root@server:~# iptables-nft -L -v
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
root@server:~# iptables-legacy -L -v
Chain INPUT (policy ACCEPT 2628K packets, 266M bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 2316K packets, 237M bytes)
pkts bytes target prot opt in out source destination
root@server:~# nft list ruleset
table inet filter {
chain input {
type filter hook input priority filter; policy accept;
iif "lo" accept
ip protocol icmp icmp type echo-request limit rate over
10/second burst 4 packets drop
ip6 nexthdr ipv6-icmp icmpv6 type echo-request limit rate over
10/second burst 4 packets drop
ct state established,related accept
ip6 nexthdr ipv6-icmp icmpv6 type { destination-unreachable,
packet-too-big, time-exceeded, parameter-problem, echo-request,
mld-listener-query, mld-listener-report, mld-listener-done, nd-router-solicit,
nd-rou
ip protocol icmp icmp type { destination-unreachable,
echo-request, router-advertisement, router-solicitation, time-exceeded,
parameter-problem } accept
ip protocol igmp accept
tcp dport 22 accept
tcp dport 80 accept
tcp dport 443 accept
tcp dport 25 accept
tcp dport 587 accept
tcp dport 465 accept
tcp dport 110 accept
tcp dport 995 accept
tcp dport 143 accept
tcp dport 993 accept
counter packets 355822 bytes 22452621 drop
}
chain output {
type filter hook output priority filter; policy accept;
}
chain forward {
type filter hook forward priority filter; policy drop;
}
}
>
>> However, then there are kernel modules loaded when looking for „net OR
>> filter OR fire OR ip“ as followed (of which I assume are just loaded as part
>> of the default base system but not doing anyhting - how to be sure of it):
>> root@server:~# lsmod |egrep -i "net|filter|fire|ip"
>> inet_diag 28672 1 tcp_diag
>> iptable_nat16384 0
>> nf_nat 49152 1 iptable_nat
>> iptable_filter 16384 0
>> nf_defrag_ipv6 20480 1 nf_conntrack
>> nf_defrag_ipv4 16384 1 nf_conntrack
>> nfnetlink 20480 1 nf_tables
>> ip_tables 32768 2 iptable_filter,iptable_nat
>> x_tables 53248 3 iptable_filter,ip_tables,iptable_nat
>> ipv6 557056 20
> Just for reference:
>
> # lsmod | grep -E '^nf' | wc -l
> 34
>
>
> so there may be a lot more, which your grep filter hid from you.
Good point, thx! Indeed modules where missed with my simple string grep….
root@server:~# lsmod | grep -E '^nf'
nf_nat 49152 1 iptable_nat
nft_ct 20480 1
nf_conntrack 143360 2 nf_nat,nft_ct
nf_defrag_ipv6 20480 1 nf_conntrack
nf_defrag_ipv4 16384 1 nf_conntrack
nft_limit 16384 2
nf_tables 237568 93 nft_ct,nft_limit
nfnetlink 20480 1 nf_tables
>
> I'm also noticing that fwbuilder, my tool of choice, seems to be scratching
> on the border line between "stable" and "legacy"...
>
> Cheers,
>
> Arno
Re: Network tcp/iptables issue with XRDP
> On 24 Oct 2023, at 16:33, Max Nikulin wrote: > > On 24/10/2023 19:04, Henggi wrote: >> - iptables on server are cleared/open (firewalld or other firewall >> frameworks are not used/installed). > > Nowadays nft or iptables is not the only option to drop packets. Another one > is eBPF used e.g. by systemd. > > I have the following link in my notes, but I have not tried the suggested tool > > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/capturing-network-packets_configuring-and-managing-networking > > 44.1. Using xdpdump to capture network packets including packets dropped by > XDP programs > Thanks… but xdpdump shows the same as tcpdump did before… which is printing all the tcp syn (seq) requests but NO tcp-ack (confirming tcp connection) going back from xrdp to client… root@server:~# xdpdump -i eth0 -w - | tcpdump -r - port not 22 and host 192.168.178.62 WARNING: Specified interface does not have an XDP program loaded, capturing in legacy mode! listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes reading from file -, link-type EN10MB (Ethernet), snapshot length 262144 17:49:41.215494 IP client.53915 > server.ms-wbt-server: Flags [S], seq 589092217, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4267962705 ecr 0,sackOK,eol], length 0 17:49:41.316585 IP client.53915 > server.ms-wbt-server: Flags [S], seq 589092217, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4267962806 ecr 0,sackOK,eol], length 0 17:49:41.417424 IP client.53915 > server.ms-wbt-server: Flags [S], seq 589092217, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4267962907 ecr 0,sackOK,eol], length 0 17:49:41.517658 IP client.53915 > server.ms-wbt-server: Flags [S], seq 589092217, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4267963008 ecr 0,sackOK,eol], length 0 17:49:41.618765 IP client.53915 > server.ms-wbt-server: Flags [S], seq 589092217, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4267963109 ecr 0,sackOK,eol], length 0 17:49:41.719768 IP client.53915 > server.ms-wbt-server: Flags [S], seq 589092217, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4267963210 ecr 0,sackOK,eol], length 0 17:49:41.920573 IP client.53915 > server.ms-wbt-server: Flags [S], seq 589092217, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4267963410 ecr 0,sackOK,eol], length 0 17:49:42.319819 IP client.53915 > server.ms-wbt-server: Flags [S], seq 589092217, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4267963810 ecr 0,sackOK,eol], length 0 17:49:43.120533 IP client.53915 > server.ms-wbt-server: Flags [S], seq 589092217, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4267964610 ecr 0,sackOK,eol], length 0 17:49:44.720237 IP client.53915 > server.ms-wbt-server: Flags [S], seq 589092217, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4267966210 ecr 0,sackOK,eol], length 0 17:49:47.921455 IP client.53915 > server.ms-wbt-server: Flags [S], seq 589092217, win 65535, options [mss 1460,sackOK,eol], length 0 ^C 74 packets captured 0 packets dropped by kernel …while doing the same test from localhost (via „lo“ interface) shows tcp syn + ack correctly as expected: root@server:~# xdpdump -i lo -w - | tcpdump -r - port 3389 WARNING: Specified interface does not have an XDP program loaded, capturing in legacy mode! listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes reading from file -, link-type EN10MB (Ethernet), snapshot length 262144 17:57:51.931307 IP server.33514 > server.ms-wbt-server: Flags [S], seq 2728692331, win 65495, options [mss 65495,sackOK,TS val 712510363 ecr 0,nop,wscale 7], length 0 17:57:51.931365 IP server.ms-wbt-server > server.33514: Flags [S.], seq 2692007133, ack 2728692332, win 65483, options [mss 65495,sackOK,TS val 712510363 ecr 712510363,nop,wscale 7], length 0 17:57:51.931419 IP server.33514 > server.ms-wbt-server: Flags [.], ack 1, win 512, options [nop,nop,TS val 712510363 ecr 712510363], length 0 17:57:51.931734 IP server.33514 > server.ms-wbt-server: Flags [F.], seq 1, ack 1, win 512, options [nop,nop,TS val 712510363 ecr 712510363], length 0 17:57:51.933177 IP server.ms-wbt-server > server.33514: Flags [.], ack 2, win 512, options [nop,nop,TS val 712510365 ecr 712510363], length 0 17:57:51.935228 IP server.ms-wbt-server > server.33514: Flags [F.], seq 1, ack 2, win 512, options [nop,nop,TS val 712510367 ecr 712510363], length 0 17:57:51.935287 IP server.33514 > server.ms-wbt-server: Flags [.], ack 2, win 512, options [nop,nop,TS val 712510367 ecr 712510367], length 0 ^C 7 packets captured 0 packets dropped by kernel
Re: Network tcp/iptables issue with XRDP
> On 24 Oct 2023, at 17:22, Charles Curley > wrote: > > On Tue, 24 Oct 2023 15:30:14 +0200 > Arno Lehmann wrote: > >> Recently I encountered something similar, and my usual test for local >> firewall being active, >> >> iptables -L -n >> >> came back with policies "accept" all over the place, and no >> particular rules. >> >> Took me a while to understand that firewalld can still do its job. firewalld not installed: root@server:~# systemctl status firewalld Unit firewalld.service could not be found. > > > > -- > Does anybody read signatures any more? > > https://charlescurley.com > https://charlescurley.com/blog/ >
Re: Network tcp/iptables issue with XRDP
Hello, Am 24.10.2023 um 16:19 schrieb Henggi: ... As I mentioned in my 1st email, I think (afaik) that no other netfitler module/service is running. My anecdote was intended to illustrate that beliefs are not really good tools to diagnose problems ;-) root@server:~# systemctl status firewalld Unit firewalld.service could not be found. Ok, so whatever it is, it's not firewalld managed on the local host. What does nft list ruleset show? (Unfortunately, this is all I know about netfilter diagnostics... and I couldn't even get this far without internet search engine :-) However, then there are kernel modules loaded when looking for „net OR filter OR fire OR ip“ as followed (of which I assume are just loaded as part of the default base system but not doing anyhting - how to be sure of it): root@server:~# lsmod |egrep -i "net|filter|fire|ip" inet_diag 28672 1 tcp_diag iptable_nat16384 0 nf_nat 49152 1 iptable_nat iptable_filter 16384 0 nf_defrag_ipv6 20480 1 nf_conntrack nf_defrag_ipv4 16384 1 nf_conntrack nfnetlink 20480 1 nf_tables ip_tables 32768 2 iptable_filter,iptable_nat x_tables 53248 3 iptable_filter,ip_tables,iptable_nat ipv6 557056 20 Just for reference: # lsmod | grep -E '^nf' | wc -l 34 so there may be a lot more, which your grep filter hid from you. I'm also noticing that fwbuilder, my tool of choice, seems to be scratching on the border line between "stable" and "legacy"... Cheers, Arno -- Arno Lehmann IT-Service Lehmann Sandstr. 6, 49080 Osnabrück
Re: Network tcp/iptables issue with XRDP
On Tue, 24 Oct 2023 15:30:14 +0200 Arno Lehmann wrote: > Recently I encountered something similar, and my usual test for local > firewall being active, > > iptables -L -n > > came back with policies "accept" all over the place, and no > particular rules. > > Took me a while to understand that firewalld can still do its job. -- Does anybody read signatures any more? https://charlescurley.com https://charlescurley.com/blog/
Re: Network tcp/iptables issue with XRDP
> On 24 Oct 2023, at 15:30, Arno Lehmann wrote: > > Hi Henggi, all, Hi Arno, thanks for your reply! > > Am 24.10.2023 um 14:04 schrieb Henggi: >> Hi list, >> Completely stuck here, any clue appreciated! >> Trying to bring up XRDP service on Debian 11-bullsyeye (arm64, incl. >> backports, fully up-to-date) which is only listening on „lo“ interface (not >> eth0) even netstat indicates otherwise: >> -> incoming tcp syn/ack on localhost interfact (lo) works fine >> -> incoming tcp syn/ack on eth0 interface seems not to reach app listening >> process (while other services on same host are working just fine via the >> network - so it’s not an „physical" network issue). >> -> iptables are cleared and not aware of any other netfilter running… > > I suggest to verify the other netfiler options. > > Recently I encountered something similar, and my usual test for local > firewall being active, > > iptables -L -n > > came back with policies "accept" all over the place, and no particular rules. > > Took me a while to understand that firewalld can still do its job. > > So, probably useful to check with > > systemctl status firewalld As I mentioned in my 1st email, I think (afaik) that no other netfitler module/service is running. root@server:~# systemctl status firewalld Unit firewalld.service could not be found. However, then there are kernel modules loaded when looking for „net OR filter OR fire OR ip“ as followed (of which I assume are just loaded as part of the default base system but not doing anyhting - how to be sure of it): root@server:~# lsmod |egrep -i "net|filter|fire|ip" inet_diag 28672 1 tcp_diag iptable_nat16384 0 nf_nat 49152 1 iptable_nat iptable_filter 16384 0 nf_defrag_ipv6 20480 1 nf_conntrack nf_defrag_ipv4 16384 1 nf_conntrack nfnetlink 20480 1 nf_tables ip_tables 32768 2 iptable_filter,iptable_nat x_tables 53248 3 iptable_filter,ip_tables,iptable_nat ipv6 557056 20 > > and use firewallcmd in an appropriate manner, if you find that to be active. > > Good luck! > > Arno > > -- > Arno Lehmann > > IT-Service Lehmann > Sandstr. 6, 49080 Osnabrück >
Re: Network tcp/iptables issue with XRDP
On 24/10/2023 19:04, Henggi wrote: - iptables on server are cleared/open (firewalld or other firewall frameworks are not used/installed). Nowadays nft or iptables is not the only option to drop packets. Another one is eBPF used e.g. by systemd. I have the following link in my notes, but I have not tried the suggested tool https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/capturing-network-packets_configuring-and-managing-networking 44.1. Using xdpdump to capture network packets including packets dropped by XDP programs
Re: Network tcp/iptables issue with XRDP
> On 24 Oct 2023, at 15:14, Dan Ritter wrote:
>
> Henggi wrote:
>>
>>> On 24 Oct 2023, at 14:46, Dan Ritter wrote:
>>>
>> - when using „port=tcp://:3389“ in xrdp.ini:
>> root@server:~# ss -tlnp | grep 3389
>> LISTEN 0 2 0.0.0.0:3389 0.0.0.0:*
>> users:(("xrdp",pid=96436,fd=11))
>>
>> - when using using „port=3389“ in xrdp.ini:
>> root@mail:~# ss -tlnp | grep 3389
>> LISTEN 0 2 *:3389 *:*
>> users:(("xrdp",pid=202831,fd=11))
>
>
>> -> either way behaviour is the same, tcp connection works from localhost but
>> not from other host.
>
> Note that "0.0.0.0" means IPv4 only, and * means v4 and v6 will
> be accepted.
I know… however, the reason why I pointed our the port config from xrdp.ini in
1st place, is that (according to "netstat -tulpn“) xrdp configured for * (to
listen on tcp4 && tcp6) does ONLY listen on tcp6 but not tcp4.
- when using using „port=3389“ in xrdp.ini:
root@server:~# netstat -tulpn |grep -i rdp
tcp6 0 0 :::3389 :::*LISTEN
202831/xrdp
tcp6 0 0 127.0.0.1:3350 :::*LISTEN
202821/xrdp-sesman
- when using „port=tcp://:3389“ in xrdp.ini:
root@server:~# netstat -tulpn |grep -i rdp
tcp0 0 0.0.0.0:33890.0.0.0:* LISTEN
204201/xrdp
tcp6 0 0 127.0.0.1:3350 :::*LISTEN
204191/xrdp-sesman
>
> Given the information that you have supplied, there is a packet
> dropping mechanism of some kind between your process on the
> other machine and this one. It could be on that other machine,
> on this machine, or in some network device between them.
My assumption too (unless it’s a weird bug within used pkg version of xrdp)…
thats why i provided evidence of the cleared iptables & tcpdump before
How to figure out where exactly the connection gets dropped (e.g. which process
or kernel module is dropping it)?
>
> Can you show that other ports on this machine are reachable from
> the other one? Say, 22?
client:~ user$ nc -zv 192.168.178.156 22
Connection to 192.168.178.156 port 22 [tcp/ssh] succeeded!
client:~ user$ nc -zv 192.168.178.156 25
Connection to 192.168.178.156 port 25 [tcp/smtp] succeeded!
client:~ user$ nc -zv 192.168.178.156 3389
nc: connectx to 192.168.178.156 port 3389 (tcp) failed: Operation timed out
> -dsr-
Re: Network tcp/iptables issue with XRDP
Hi Henggi, all, Am 24.10.2023 um 14:04 schrieb Henggi: Hi list, Completely stuck here, any clue appreciated! Trying to bring up XRDP service on Debian 11-bullsyeye (arm64, incl. backports, fully up-to-date) which is only listening on „lo“ interface (not eth0) even netstat indicates otherwise: -> incoming tcp syn/ack on localhost interfact (lo) works fine -> incoming tcp syn/ack on eth0 interface seems not to reach app listening process (while other services on same host are working just fine via the network - so it’s not an „physical" network issue). -> iptables are cleared and not aware of any other netfilter running… I suggest to verify the other netfiler options. Recently I encountered something similar, and my usual test for local firewall being active, iptables -L -n came back with policies "accept" all over the place, and no particular rules. Took me a while to understand that firewalld can still do its job. So, probably useful to check with systemctl status firewalld and use firewallcmd in an appropriate manner, if you find that to be active. Good luck! Arno -- Arno Lehmann IT-Service Lehmann Sandstr. 6, 49080 Osnabrück
Re: Network tcp/iptables issue with XRDP
Henggi wrote:
>
> > On 24 Oct 2023, at 14:46, Dan Ritter wrote:
> >
> - when using „port=tcp://:3389“ in xrdp.ini:
> root@server:~# ss -tlnp | grep 3389
> LISTEN 0 2 0.0.0.0:3389 0.0.0.0:*
> users:(("xrdp",pid=96436,fd=11))
>
> - when using using „port=3389“ in xrdp.ini:
> root@mail:~# ss -tlnp | grep 3389
> LISTEN 0 2 *:3389 *:*
> users:(("xrdp",pid=202831,fd=11))
> -> either way behaviour is the same, tcp connection works from localhost but
> not from other host.
Note that "0.0.0.0" means IPv4 only, and * means v4 and v6 will
be accepted.
Given the information that you have supplied, there is a packet
dropping mechanism of some kind between your process on the
other machine and this one. It could be on that other machine,
on this machine, or in some network device between them.
Can you show that other ports on this machine are reachable from
the other one? Say, 22?
-dsr-
Re: Network tcp/iptables issue with XRDP
> On 24 Oct 2023, at 14:46, Dan Ritter wrote:
>
> Henggi wrote:
>> Hi list,
>>
>> Completely stuck here, any clue appreciated!
>>
>> — by default xrdp.ini listening config is set to "port=3389“ (which is
>> expected to listen on ipv4 && ipv6 in parallel as I understand). However
>> using that seems only to listen on tcp4 according to „netstat“
>> — therefore changed listening config to "port=tcp://:3389“ (which is
>> exprected to listen on ip4/tcp only)… which seems todo that as per netstat
>> output shown above
>
>
> man xrdp,ini:
>
> address=ip address
>Specify xrdp listening address. If not specified, defaults to 0.0.0.0 (all
> interfaces).
Never had any address specified in xrdp.ini… so it’s expected to listen to
default 0.0.0.0 (all interfaces) which it did already before acc. to netstat
(as mentioned in prev. email)
> ...
> port=port
>Specify TCP port to listen on for incoming connections. The default for
> RDP is 3389.
>
>
> Remove or comment out all address and port lines from the config and
> restart xrdp.
>
> Show us the TCP listener on the machine after that:
>
> ss -tlnp | grep 3389
- when using „port=tcp://:3389“ in xrdp.ini:
root@server:~# ss -tlnp | grep 3389
LISTEN 0 2 0.0.0.0:3389 0.0.0.0:*
users:(("xrdp",pid=96436,fd=11))
- when using using „port=3389“ in xrdp.ini:
root@mail:~# ss -tlnp | grep 3389
LISTEN 0 2 *:3389 *:*
users:(("xrdp",pid=202831,fd=11))
-> either way behaviour is the same, tcp connection works from localhost but
not from other host.
>
>
> Then try connecting from another machine.
>
> -dsr-
Re: Network tcp/iptables issue with XRDP
Henggi wrote: > Hi list, > > Completely stuck here, any clue appreciated! > > — by default xrdp.ini listening config is set to "port=3389“ (which is > expected to listen on ipv4 && ipv6 in parallel as I understand). However > using that seems only to listen on tcp4 according to „netstat“ > — therefore changed listening config to "port=tcp://:3389“ (which is > exprected to listen on ip4/tcp only)… which seems todo that as per netstat > output shown above man xrdp,ini: address=ip address Specify xrdp listening address. If not specified, defaults to 0.0.0.0 (all interfaces). ... port=port Specify TCP port to listen on for incoming connections. The default for RDP is 3389. Remove or comment out all address and port lines from the config and restart xrdp. Show us the TCP listener on the machine after that: ss -tlnp | grep 3389 Then try connecting from another machine. -dsr-
