Bug#893959: iproute2: tc show option in tc class show and tc filter show not working correctly

[Václav Zindulka]

On Thu, Mar 29, 2018 at 4:09 PM, Luca Boccassi  wrote:

> Control: tags -1 upstream
>
> On Thu, 2018-03-29 at 14:31 +0200, Václav Zindulka wrote:
> > Hello,
> >
> > thank you both for tips. I'll fix it in program for now. I may try to
> > have
> > a look at hfsc source one day, but now I have to finish my project
> > and get
> > it to production state.
> >
> >
> > 
> > S pozdravem / Best Regards
> >
> > Vaclav Zindulka
>
> You can also try to ask the same question on the netdev kernel mailing
> list, as Stephen said this is probably due to code in the kernel.
>
>
I know, but he also mentioned that nobody uses it so it is buggy. So for
now I'll leave this just for reference in case anybody else comes across
this issue. At the bottom of message is tc -g class show - I hope it won't
break formatting too much.

As I mentioned, I may try to look at it, but now I can't leave one project
to fix other. If anybody is willing to investigate it, so be it. My current
project is in far worse state so I need to prioritize it as much as
possible. This bug means just ignoring second occurrence of tc class since
all values are the same. I can live with it for now.

I admit I don't have right skills to assist anybody with patching and
testing those patches. I need to acquire those skills and this could be the
right way.

Thank you for your time anyway.

Have a nice day :-)

+---(2064:) hfsc
||   Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
||   backlog 0b 0p requeues 0
||
|+---(2064:) hfsc ls m1 2048Kbit d 20.0s m2 512Kbit ul m1 0bit d
0us m2 3072Kbit
||Sent 52121422 bytes 38713 pkt (dropped 6, overlimits
0 requeues 0)
||backlog 79850b 54p requeues 0
||
|+---(2064:2) hfsc ls m1 0bit d 0us m2 1850Kbit ul m1 0bit d 0us m2
4507Kbit
|| Sent 286644532 bytes 202823 pkt (dropped 1, overlimits 0
requeues 0)
|| backlog 39826b 27p requeues 0
||
|+---(2064:3) hfsc ls m1 800Kbit d 0us m2 300720bit ul m1 0bit d 0us m2
4507Kbit
|| Sent 47629230 bytes 39145 pkt (dropped 5, overlimits 0
requeues 0)
|| backlog 71800b 43p requeues 0
||
|+---(2064:) hfsc ls m1 2048Kbit d 20.0s m2 512Kbit ul m1 0bit d
0us m2 3072Kbit
||Sent 52121422 bytes 38713 pkt (dropped 6, overlimits
0 requeues 0)
||backlog 79850b 54p requeues 0
||
|+---(2064:2) hfsc ls m1 0bit d 0us m2 1850Kbit ul m1 0bit d 0us m2
4507Kbit
|| Sent 286644532 bytes 202823 pkt (dropped 1, overlimits 0
requeues 0)
|| backlog 39826b 27p requeues 0
||
|+---(2064:3) hfsc ls m1 800Kbit d 0us m2 300720bit ul m1 0bit d 0us m2
4507Kbit
|  Sent 47629230 bytes 39145 pkt (dropped 5, overlimits 0
requeues 0)
|  backlog 71800b 43p requeues 0
|
+---(2064:) hfsc
|Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
|backlog 0b 0p requeues 0
|
+---(1:) hfsc
 |Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
 |backlog 0b 0p requeues 0
 |
 +---(1:1) hfsc ls m1 0bit d 0us m2 13107Kbit ul m1 0bit d 0us m2
13107Kbit
  | Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
  | backlog 0b 0p requeues 0
  |
  +---(1:) hfsc ls m1 0bit d 0us m2 250Kbit ul m1 0bit d 0us m2
2500Kbit
  | Sent 98492 bytes 102 pkt (dropped 0, overlimits 0
requeues 0)
  | backlog 0b 0p requeues 0
  |
  +---(1:2064) hfsc ls m1 0bit d 0us m2 4Mbit ul m1 0bit d 0us m2
5Mbit
Sent 386395184 bytes 280681 pkt (dropped 12,
overlimits 0 requeues 0)
backlog 187Kb 124p requeues 0



S pozdravem / Best Regards

Vaclav Zindulka

Bug#893959: iproute2: tc show option in tc class show and tc filter show not working correctly

[Luca Boccassi]

Control: tags -1 upstream

On Thu, 2018-03-29 at 14:31 +0200, Václav Zindulka wrote:
> Hello,
> 
> thank you both for tips. I'll fix it in program for now. I may try to
> have
> a look at hfsc source one day, but now I have to finish my project
> and get
> it to production state.
> 
> 
> 
> S pozdravem / Best Regards
> 
> Vaclav Zindulka

You can also try to ask the same question on the netdev kernel mailing
list, as Stephen said this is probably due to code in the kernel.

> 
> On Wed, Mar 28, 2018 at 1:32 AM, Stephen Hemminger <
> step...@networkplumber.org> wrote:
> 
> > On Mon, 26 Mar 2018 13:34:39 +0100
> > Luca Boccassi  wrote:
> > 
> > > On Sat, 2018-03-24 at 14:43 +0100, Vaclav Zindulka wrote:
> > > > Package: iproute2
> > > > Version: 4.14.1-1~bpo9+1
> > > > Severity: normal
> > > > 
> > > > Dear Maintainer,
> > > > 
> > > > I found some problems in tc class show and tc filter show
> > > > commands.
> > > > We
> > > > use tc in large scale (thousands of subnets) on multiple
> > > > servers.
> > > > Whole
> > > > shaping system is udergoing large overview and I'm rewriting it
> > > > almost
> > > > from the scratch. Tc rules are one of key elements so I started
> > > > to
> > > > check
> > > > and change its whole structure. Until now we had everything in
> > > > parent
> > > > qdisc 1:0 (all tc filter rules, tc classes). Now I decided to
> > > > change
> > > > it,
> > > > so a lot of classes have its own hfsc qdiscs and in this qdiscs
> > > > I
> > > > create
> > > > whole new structure of tc classes, tc filters a tc qdiscs.
> > > > Let's say
> > > > it
> > > > is a tree within a tree. I can provide simple (prototype) bash
> > > > script,
> > > > which led to my discovery. (https://pastebin.com/A57x8fq5 -
> > > > comments
> > > > are
> > > > in czech since I didn't plan to release it anywhere :-) - I can
> > > > provide
> > > > further info if needed.
> > > > 
> > > > Problem is very annoying since I need to get
> > > > stats from tc -s class show dev ...etc.
> > > > 
> > > > 1. First problem - tc class show dev enp0s31f6 prints whole
> > > > structure
> > > > two times. I compared it with "sudo tc -s class show dev
> > > > enp0s31f6" |
> > > > grep
> > > > -c "class" vs "sudo tc -s class show dev enp0s31f6" | sort |
> > > > uniq |
> > > > grep
> > > > -c "class" which produces about a half of records. Reference
> > > > class
> > > > 2000: (produced by my script 2000 - 4fff) is shown twice
> > > > (same
> > > > for
> > > > 2000: root). I'm not sure about behavior of executing command
> > > > twice
> > > > in this case but it can't be defined twice for sure. Tc is
> > > > veeery
> > > > picky
> > > > :-)
> > > > 
> > > > 2. Second problem - tc filter show dev enp0s31f6 show just the
> > > > parent
> > > > 1:0 rules. This one is not so critical, since I don't read
> > > > stats
> > > > regularly, but for debugging it is very inconvenient since one
> > > > has to
> > > > know classId of whole parent (let's say 2000:0 qdisc in which
> > > > all
> > > > those
> > > > filters reside). When I execute tc filter show dev enp0s31f6
> > > > parent
> > > > 2000: (including colon :-) I get all those defined filter rules
> > > > and
> > > > hash
> > > > tables. But not with base command.
> > > > 
> > > > I discovered this on version 4.9 from stable stretch
> > > > repository. I
> > > > tried
> > > > to install backport version 4.14 but resutls were exactly the
> > > > same. I
> > > > didn't try to reboot after upgrade yet tho. I'll do this after
> > > > finishing
> > > > this bugreport. I didn't try to compile newest version from
> > > > source
> > > > since
> > > > I need some kind of stable solution to put it on 21 servers.
> > > > 
> > > > What I expected was to see defined classes just once. And to
> > > > see tc
> > > > filter rules with base show command without specifying parent's
> > > > classid
> > > > for every hfsc qdisc containing additional rules and hash
> > > > tables.
> > > 
> > > Hi,
> > > 
> > > Since you mentioned it's working in the same way in 4.9 to 4.14
> > > it
> > > doesn't look like it's a regression, so a better place to ask
> > > would
> > > probably be netdev: http://vger.kernel.org/vger-lists.html#netdev
> > > 
> > > Stephen, any comment on the behaviour described above?
> > > 
> > 
> > No idea, the print code in iproute2 is just taking what kernel
> > passes back and displays. Most likely it is in hfsc code which to
> > be honest
> > is not widely used by the kernel maintainers. I wouldn't be
> > surprised if it
> > had bugs.
> > 

-- 
Kind regards,
Luca Boccassi

signature.asc
Description: This is a digitally signed message part

Bug#893959: iproute2: tc show option in tc class show and tc filter show not working correctly

[Václav Zindulka]

Hello,

thank you both for tips. I'll fix it in program for now. I may try to have
a look at hfsc source one day, but now I have to finish my project and get
it to production state.



S pozdravem / Best Regards

Vaclav Zindulka


On Wed, Mar 28, 2018 at 1:32 AM, Stephen Hemminger <
step...@networkplumber.org> wrote:

> On Mon, 26 Mar 2018 13:34:39 +0100
> Luca Boccassi  wrote:
>
> > On Sat, 2018-03-24 at 14:43 +0100, Vaclav Zindulka wrote:
> > > Package: iproute2
> > > Version: 4.14.1-1~bpo9+1
> > > Severity: normal
> > >
> > > Dear Maintainer,
> > >
> > > I found some problems in tc class show and tc filter show commands.
> > > We
> > > use tc in large scale (thousands of subnets) on multiple servers.
> > > Whole
> > > shaping system is udergoing large overview and I'm rewriting it
> > > almost
> > > from the scratch. Tc rules are one of key elements so I started to
> > > check
> > > and change its whole structure. Until now we had everything in parent
> > > qdisc 1:0 (all tc filter rules, tc classes). Now I decided to change
> > > it,
> > > so a lot of classes have its own hfsc qdiscs and in this qdiscs I
> > > create
> > > whole new structure of tc classes, tc filters a tc qdiscs. Let's say
> > > it
> > > is a tree within a tree. I can provide simple (prototype) bash
> > > script,
> > > which led to my discovery. (https://pastebin.com/A57x8fq5 - comments
> > > are
> > > in czech since I didn't plan to release it anywhere :-) - I can
> > > provide
> > > further info if needed.
> > >
> > > Problem is very annoying since I need to get
> > > stats from tc -s class show dev ...etc.
> > >
> > > 1. First problem - tc class show dev enp0s31f6 prints whole structure
> > > two times. I compared it with "sudo tc -s class show dev enp0s31f6" |
> > > grep
> > > -c "class" vs "sudo tc -s class show dev enp0s31f6" | sort | uniq |
> > > grep
> > > -c "class" which produces about a half of records. Reference class
> > > 2000: (produced by my script 2000 - 4fff) is shown twice (same
> > > for
> > > 2000: root). I'm not sure about behavior of executing command twice
> > > in this case but it can't be defined twice for sure. Tc is veeery
> > > picky
> > > :-)
> > >
> > > 2. Second problem - tc filter show dev enp0s31f6 show just the parent
> > > 1:0 rules. This one is not so critical, since I don't read stats
> > > regularly, but for debugging it is very inconvenient since one has to
> > > know classId of whole parent (let's say 2000:0 qdisc in which all
> > > those
> > > filters reside). When I execute tc filter show dev enp0s31f6 parent
> > > 2000: (including colon :-) I get all those defined filter rules and
> > > hash
> > > tables. But not with base command.
> > >
> > > I discovered this on version 4.9 from stable stretch repository. I
> > > tried
> > > to install backport version 4.14 but resutls were exactly the same. I
> > > didn't try to reboot after upgrade yet tho. I'll do this after
> > > finishing
> > > this bugreport. I didn't try to compile newest version from source
> > > since
> > > I need some kind of stable solution to put it on 21 servers.
> > >
> > > What I expected was to see defined classes just once. And to see tc
> > > filter rules with base show command without specifying parent's
> > > classid
> > > for every hfsc qdisc containing additional rules and hash tables.
> >
> > Hi,
> >
> > Since you mentioned it's working in the same way in 4.9 to 4.14 it
> > doesn't look like it's a regression, so a better place to ask would
> > probably be netdev: http://vger.kernel.org/vger-lists.html#netdev
> >
> > Stephen, any comment on the behaviour described above?
> >
>
> No idea, the print code in iproute2 is just taking what kernel
> passes back and displays. Most likely it is in hfsc code which to be honest
> is not widely used by the kernel maintainers. I wouldn't be surprised if it
> had bugs.
>

Bug#893959: iproute2: tc show option in tc class show and tc filter show not working correctly

[Stephen Hemminger]

On Mon, 26 Mar 2018 13:34:39 +0100
Luca Boccassi  wrote:

> On Sat, 2018-03-24 at 14:43 +0100, Vaclav Zindulka wrote:
> > Package: iproute2
> > Version: 4.14.1-1~bpo9+1
> > Severity: normal
> > 
> > Dear Maintainer,
> > 
> > I found some problems in tc class show and tc filter show commands.
> > We
> > use tc in large scale (thousands of subnets) on multiple servers.
> > Whole
> > shaping system is udergoing large overview and I'm rewriting it
> > almost
> > from the scratch. Tc rules are one of key elements so I started to
> > check
> > and change its whole structure. Until now we had everything in parent
> > qdisc 1:0 (all tc filter rules, tc classes). Now I decided to change
> > it,
> > so a lot of classes have its own hfsc qdiscs and in this qdiscs I
> > create
> > whole new structure of tc classes, tc filters a tc qdiscs. Let's say
> > it
> > is a tree within a tree. I can provide simple (prototype) bash
> > script,
> > which led to my discovery. (https://pastebin.com/A57x8fq5 - comments
> > are
> > in czech since I didn't plan to release it anywhere :-) - I can
> > provide
> > further info if needed.
> > 
> > Problem is very annoying since I need to get
> > stats from tc -s class show dev ...etc. 
> > 
> > 1. First problem - tc class show dev enp0s31f6 prints whole structure
> > two times. I compared it with "sudo tc -s class show dev enp0s31f6" |
> > grep
> > -c "class" vs "sudo tc -s class show dev enp0s31f6" | sort | uniq |
> > grep
> > -c "class" which produces about a half of records. Reference class
> > 2000: (produced by my script 2000 - 4fff) is shown twice (same
> > for
> > 2000: root). I'm not sure about behavior of executing command twice
> > in this case but it can't be defined twice for sure. Tc is veeery
> > picky
> > :-)
> > 
> > 2. Second problem - tc filter show dev enp0s31f6 show just the parent
> > 1:0 rules. This one is not so critical, since I don't read stats
> > regularly, but for debugging it is very inconvenient since one has to
> > know classId of whole parent (let's say 2000:0 qdisc in which all
> > those
> > filters reside). When I execute tc filter show dev enp0s31f6 parent
> > 2000: (including colon :-) I get all those defined filter rules and
> > hash
> > tables. But not with base command.
> > 
> > I discovered this on version 4.9 from stable stretch repository. I
> > tried
> > to install backport version 4.14 but resutls were exactly the same. I
> > didn't try to reboot after upgrade yet tho. I'll do this after
> > finishing
> > this bugreport. I didn't try to compile newest version from source
> > since
> > I need some kind of stable solution to put it on 21 servers.
> > 
> > What I expected was to see defined classes just once. And to see tc
> > filter rules with base show command without specifying parent's
> > classid
> > for every hfsc qdisc containing additional rules and hash tables.  
> 
> Hi,
> 
> Since you mentioned it's working in the same way in 4.9 to 4.14 it
> doesn't look like it's a regression, so a better place to ask would
> probably be netdev: http://vger.kernel.org/vger-lists.html#netdev
> 
> Stephen, any comment on the behaviour described above?
> 

No idea, the print code in iproute2 is just taking what kernel
passes back and displays. Most likely it is in hfsc code which to be honest
is not widely used by the kernel maintainers. I wouldn't be surprised if it
had bugs.


pgpC3PGwMtHwY.pgp
Description: OpenPGP digital signature

Bug#893959: iproute2: tc show option in tc class show and tc filter show not working correctly

[Luca Boccassi]

On Sat, 2018-03-24 at 14:43 +0100, Vaclav Zindulka wrote:
> Package: iproute2
> Version: 4.14.1-1~bpo9+1
> Severity: normal
> 
> Dear Maintainer,
> 
> I found some problems in tc class show and tc filter show commands.
> We
> use tc in large scale (thousands of subnets) on multiple servers.
> Whole
> shaping system is udergoing large overview and I'm rewriting it
> almost
> from the scratch. Tc rules are one of key elements so I started to
> check
> and change its whole structure. Until now we had everything in parent
> qdisc 1:0 (all tc filter rules, tc classes). Now I decided to change
> it,
> so a lot of classes have its own hfsc qdiscs and in this qdiscs I
> create
> whole new structure of tc classes, tc filters a tc qdiscs. Let's say
> it
> is a tree within a tree. I can provide simple (prototype) bash
> script,
> which led to my discovery. (https://pastebin.com/A57x8fq5 - comments
> are
> in czech since I didn't plan to release it anywhere :-) - I can
> provide
> further info if needed.
> 
> Problem is very annoying since I need to get
> stats from tc -s class show dev ...etc. 
> 
> 1. First problem - tc class show dev enp0s31f6 prints whole structure
> two times. I compared it with "sudo tc -s class show dev enp0s31f6" |
> grep
> -c "class" vs "sudo tc -s class show dev enp0s31f6" | sort | uniq |
> grep
> -c "class" which produces about a half of records. Reference class
> 2000: (produced by my script 2000 - 4fff) is shown twice (same
> for
> 2000: root). I'm not sure about behavior of executing command twice
> in this case but it can't be defined twice for sure. Tc is veeery
> picky
> :-)
> 
> 2. Second problem - tc filter show dev enp0s31f6 show just the parent
> 1:0 rules. This one is not so critical, since I don't read stats
> regularly, but for debugging it is very inconvenient since one has to
> know classId of whole parent (let's say 2000:0 qdisc in which all
> those
> filters reside). When I execute tc filter show dev enp0s31f6 parent
> 2000: (including colon :-) I get all those defined filter rules and
> hash
> tables. But not with base command.
> 
> I discovered this on version 4.9 from stable stretch repository. I
> tried
> to install backport version 4.14 but resutls were exactly the same. I
> didn't try to reboot after upgrade yet tho. I'll do this after
> finishing
> this bugreport. I didn't try to compile newest version from source
> since
> I need some kind of stable solution to put it on 21 servers.
> 
> What I expected was to see defined classes just once. And to see tc
> filter rules with base show command without specifying parent's
> classid
> for every hfsc qdisc containing additional rules and hash tables.

Hi,

Since you mentioned it's working in the same way in 4.9 to 4.14 it
doesn't look like it's a regression, so a better place to ask would
probably be netdev: http://vger.kernel.org/vger-lists.html#netdev

Stephen, any comment on the behaviour described above?

-- 
Kind regards,
Luca Boccassi

signature.asc
Description: This is a digitally signed message part

Bug#893959: iproute2: tc show option in tc class show and tc filter show not working correctly

[Vaclav Zindulka]

Package: iproute2
Version: 4.14.1-1~bpo9+1
Severity: normal

Dear Maintainer,

I found some problems in tc class show and tc filter show commands. We
use tc in large scale (thousands of subnets) on multiple servers. Whole
shaping system is udergoing large overview and I'm rewriting it almost
from the scratch. Tc rules are one of key elements so I started to check
and change its whole structure. Until now we had everything in parent
qdisc 1:0 (all tc filter rules, tc classes). Now I decided to change it,
so a lot of classes have its own hfsc qdiscs and in this qdiscs I create
whole new structure of tc classes, tc filters a tc qdiscs. Let's say it
is a tree within a tree. I can provide simple (prototype) bash script,
which led to my discovery. (https://pastebin.com/A57x8fq5 - comments are
in czech since I didn't plan to release it anywhere :-) - I can provide
further info if needed.

Problem is very annoying since I need to get
stats from tc -s class show dev ...etc. 

1. First problem - tc class show dev enp0s31f6 prints whole structure
two times. I compared it with "sudo tc -s class show dev enp0s31f6" | grep
-c "class" vs "sudo tc -s class show dev enp0s31f6" | sort | uniq | grep
-c "class" which produces about a half of records. Reference class
2000: (produced by my script 2000 - 4fff) is shown twice (same for
2000: root). I'm not sure about behavior of executing command twice
in this case but it can't be defined twice for sure. Tc is veeery picky
:-)

2. Second problem - tc filter show dev enp0s31f6 show just the parent
1:0 rules. This one is not so critical, since I don't read stats
regularly, but for debugging it is very inconvenient since one has to
know classId of whole parent (let's say 2000:0 qdisc in which all those
filters reside). When I execute tc filter show dev enp0s31f6 parent
2000: (including colon :-) I get all those defined filter rules and hash
tables. But not with base command.

I discovered this on version 4.9 from stable stretch repository. I tried
to install backport version 4.14 but resutls were exactly the same. I
didn't try to reboot after upgrade yet tho. I'll do this after finishing
this bugreport. I didn't try to compile newest version from source since
I need some kind of stable solution to put it on 21 servers.

What I expected was to see defined classes just once. And to see tc
filter rules with base show command without specifying parent's classid
for every hfsc qdisc containing additional rules and hash tables.


-- System Information:
Debian Release: 9.4
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-5-amd64 (SMP w/12 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages iproute2 depends on:
ii  libc62.24-11+deb9u3
ii  libdb5.3 5.3.28-12+deb9u1
ii  libelf1  0.168-1
ii  libmnl0  1.0.4-2
ii  libselinux1  2.6-3+b3

Versions of packages iproute2 recommends:
pn  libatm1   
ii  libxtables12  1.6.0+snapshot20161117-6

Versions of packages iproute2 suggests:
ii  iproute2-doc  4.14.1-1~bpo9+1

-- no debconf information