Re: Setting up vmd with veb0/vport0
It looks like vport0 is down. Add "up" to hostname.vport0 and ifconfig
vport0 up.
On Thu, 12 May 2022 at 15:40, David Demelier wrote:
> Hello,
>
> I'm trying to setup vms using the wonderful vmd and private addresses
> on 10.0.0.0 range. Following the various entries in the FAQ (faq16) and
> the examples using bridge/vether I just wanted to adapt to using
> veb/vport instead since it's designed as a newer and more performant
> replacement.
>
> I've also seen someone who managed to get it working
>
>
> https://misc.openbsd.narkive.com/nAdmGfbQ/i-can-t-get-veb-vport-to-work-with-vmd
>
> So first, I setup the interfaces:
>
> # cat /etc/hostname.veb0
> add vport0
> up
> # cat /etc/hostname.vport0
> inet 10.0.0.1 255.255.255.0
>
> I enable NAT as specified in the FAQ and numerous examples.
>
> # cat /etc/pf.conf
> set skip on lo0
>
> match in all scrub (no-df random-id max-mss 1440)
> match out on egress inet from vport0:network to any nat-to (egress)
>
> block log
> pass out quick inet
> pass in on vport0 inet
>
> Then, setting up vmd to boot an install71.iso with the appropriate tap
> interfaces:
>
> # cat /etc/vm.conf
> switch "switch0" {
> interface veb0
> }
>
> vm "vm1" {
> disk "/vm/vm1.qcow2"
> boot device cdrom
> cdrom "/vm/install71.iso"
>
> interface tap {
> switch "switch0"
> }
> }
>
> Finally, once the install is boot, I've tried adding 10.0.0.10 netmask
> 255.255.255.0 and 10.0.0.1 as gateway with no luck. The nameserver is
> copied from /etc/resolv.conf but I can't get any packet to the
> internet.
>
> (vm) #
> ping 8.8.8.8
> PING 8.8.8.8 (8.8.8.8): 56 data bytes
> ping: sendmsg: Can't assign requested address
> ping: wrote 8.8.8.8 64 chars, ret=-1
> (vm) #
> # ftp http://5.135.187.121/index.html
> Trying 5.135.187.121...
> ftp: connect: Can't assign requested address
>
> I'm sure I miss almost nothing but I can't find what.
>
> Here's the host full ifconfig
>
> lo0: flags=8049 mtu 32768
> index 4 priority 0 llprio 3
> groups: lo
> inet6 ::1 prefixlen 128
> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
> inet 127.0.0.1 netmask 0xff00
> iwx0: flags=808843
> mtu 1500
> lladdr e0:d4:64:3c:31:9c
> index 1 priority 4 llprio 3
> groups: wlan egress
> media: IEEE802.11 autoselect (VHT-MCS9 mode 11ac)
> status: active
> ieee80211: join "abc" chan 149 bssid aa:37:d8:93:98:57 82%
> wpakey wpaprotos wpa2 wpaakms psk wpaciphers ccmp wpagroupcipher ccmp
> inet 172.20.10.3 netmask 0xfff0 broadcast 172.20.10.15
> em0: flags=808843 mtu
> 1500
> lladdr 8c:8c:aa:01:7d:1f
> index 2 priority 0 llprio 3
> media: Ethernet autoselect (none)
> status: no carrier
> enc0: flags=0<>
> index 3 priority 0 llprio 3
> groups: enc
> status: active
> veb0: flags=8843
> description: switch1-switch0
> index 5 llprio 3
> groups: veb
> vport0 flags=3
> port 7 ifpriority 0 ifcost 0
> tap0 flags=3
> port 8 ifpriority 0 ifcost 0
> vlan0: flags=8002 mtu 1500
> lladdr e0:d4:64:3c:31:9c
> index 6 priority 0 llprio 3
> encap: vnetid none parent iwx0 txprio packet rxprio outer
> groups: vlan
> media: IEEE802.11 autoselect (VHT-MCS9 mode 11ac)
> status: active
> vport0: flags=8902 mtu 1500
> lladdr fe:e1:ba:d0:32:b5
> index 7 priority 0 llprio 3
> groups: vport
> inet 10.0.0.1 netmask 0xff00 broadcast 10.0.0.255
> tap0: flags=8943 mtu
> 1500
> lladdr fe:e1:ba:d1:f2:03
> description: vm1-if0-vm1
> index 8 priority 0 llprio 3
> groups: tap
> status: active
>
> Any help is appreciated.
>
> Regards,
>
> --
> David
>
>
Re: Setting up vmd with veb0/vport0
On 2022-05-12, David Demelier wrote: > (vm) # > ping 8.8.8.8 > PING 8.8.8.8 (8.8.8.8): 56 data bytes > ping: sendmsg: Can't assign requested address > ping: wrote 8.8.8.8 64 chars, ret=-1 > (vm) # > # ftp http://5.135.187.121/index.html > Trying 5.135.187.121... > ftp: connect: Can't assign requested address How do ifconfig -A and netstat -rnfinet look in the vm? > vlan0: flags=8002 mtu 1500 > lladdr e0:d4:64:3c:31:9c > index 6 priority 0 llprio 3 > encap: vnetid none parent iwx0 txprio packet rxprio outer > groups: vlan > media: IEEE802.11 autoselect (VHT-MCS9 mode 11ac) > status: active I think this isn't doing anything, right? > vport0: flags=8902 mtu 1500 > lladdr fe:e1:ba:d0:32:b5 > index 7 priority 0 llprio 3 > groups: vport > inet 10.0.0.1 netmask 0xff00 broadcast 10.0.0.255 This interface is not "up", iirc you need to do that explicitly for vport. Also check you have set sysctl net.inet.ip.forwarding in the host. -- Please keep replies on the mailing list.
Setting up vmd with veb0/vport0
Hello,
I'm trying to setup vms using the wonderful vmd and private addresses
on 10.0.0.0 range. Following the various entries in the FAQ (faq16) and
the examples using bridge/vether I just wanted to adapt to using
veb/vport instead since it's designed as a newer and more performant
replacement.
I've also seen someone who managed to get it working
https://misc.openbsd.narkive.com/nAdmGfbQ/i-can-t-get-veb-vport-to-work-with-vmd
So first, I setup the interfaces:
# cat /etc/hostname.veb0
add vport0
up
# cat /etc/hostname.vport0
inet 10.0.0.1 255.255.255.0
I enable NAT as specified in the FAQ and numerous examples.
# cat /etc/pf.conf
set skip on lo0
match in all scrub (no-df random-id max-mss 1440)
match out on egress inet from vport0:network to any nat-to (egress)
block log
pass out quick inet
pass in on vport0 inet
Then, setting up vmd to boot an install71.iso with the appropriate tap
interfaces:
# cat /etc/vm.conf
switch "switch0" {
interface veb0
}
vm "vm1" {
disk "/vm/vm1.qcow2"
boot device cdrom
cdrom "/vm/install71.iso"
interface tap {
switch "switch0"
}
}
Finally, once the install is boot, I've tried adding 10.0.0.10 netmask
255.255.255.0 and 10.0.0.1 as gateway with no luck. The nameserver is
copied from /etc/resolv.conf but I can't get any packet to the
internet.
(vm) #
ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendmsg: Can't assign requested address
ping: wrote 8.8.8.8 64 chars, ret=-1
(vm) #
# ftp http://5.135.187.121/index.html
Trying 5.135.187.121...
ftp: connect: Can't assign requested address
I'm sure I miss almost nothing but I can't find what.
Here's the host full ifconfig
lo0: flags=8049 mtu 32768
index 4 priority 0 llprio 3
groups: lo
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
inet 127.0.0.1 netmask 0xff00
iwx0: flags=808843
mtu 1500
lladdr e0:d4:64:3c:31:9c
index 1 priority 4 llprio 3
groups: wlan egress
media: IEEE802.11 autoselect (VHT-MCS9 mode 11ac)
status: active
ieee80211: join "abc" chan 149 bssid aa:37:d8:93:98:57 82%
wpakey wpaprotos wpa2 wpaakms psk wpaciphers ccmp wpagroupcipher ccmp
inet 172.20.10.3 netmask 0xfff0 broadcast 172.20.10.15
em0: flags=808843 mtu
1500
lladdr 8c:8c:aa:01:7d:1f
index 2 priority 0 llprio 3
media: Ethernet autoselect (none)
status: no carrier
enc0: flags=0<>
index 3 priority 0 llprio 3
groups: enc
status: active
veb0: flags=8843
description: switch1-switch0
index 5 llprio 3
groups: veb
vport0 flags=3
port 7 ifpriority 0 ifcost 0
tap0 flags=3
port 8 ifpriority 0 ifcost 0
vlan0: flags=8002 mtu 1500
lladdr e0:d4:64:3c:31:9c
index 6 priority 0 llprio 3
encap: vnetid none parent iwx0 txprio packet rxprio outer
groups: vlan
media: IEEE802.11 autoselect (VHT-MCS9 mode 11ac)
status: active
vport0: flags=8902 mtu 1500
lladdr fe:e1:ba:d0:32:b5
index 7 priority 0 llprio 3
groups: vport
inet 10.0.0.1 netmask 0xff00 broadcast 10.0.0.255
tap0: flags=8943 mtu
1500
lladdr fe:e1:ba:d1:f2:03
description: vm1-if0-vm1
index 8 priority 0 llprio 3
groups: tap
status: active
Any help is appreciated.
Regards,
--
David
Re: why does resolvd sort nameserver rules
William Ahern wrote: > On Wed, May 11, 2022 at 04:54:02PM +0100, james palmer wrote: > > i have a local dhcp server running which gives out three nameservers: > > > > - 192.168.0.2 (resolves some local machine names) > > - 9.9.9.9 > > - 149.112.112.112 > > > > on linux, android, and windows the local nameserver takes priority over the > > others. > > on openbsd thanks to the nameservers being sorted by ip address > > 149.112.112.112 is chosen first. > > this causes the local machine names to not resolve. > > > > is there a reason for this behaviour? > > i would expect the nameservers to be written in the order they are set as > > on the dhcp server. > > I have no direct knowledge, but judging from the code in resolvd.c > handle_route_message and cmp, the sorting is primarily driven by declared > priority of the source--resolvers associated with a higher priority source > come first, independent of IP address. However, the relative ordering > information among a set of proposed resolvers from a particular routing > message seems to be dropped on the floor, no other information is used to > sort, and the sort comparator falls back to comparing the address strings > (using strcmp) to achieve a stable sort. yes, but you missed a piece: /* Eliminate duplicates */ Duplicates should be eliminated when possible, because libc/asr only uses the first 5 addresses I think. Imagine we have a v4 announcement doing 3 addresses, and a v4 "umb" doing 2 addreses, libc will never get around to using a manual entry at the end if those dynamic dns servers are in fact not responding. I was trying to handle that kind of situation heuristically. Maybe we can avoid the address sort, but still attempt to delete duplicate addresses, to increasae the odds of reaching a manual entry. > This could be fixed, assuming the current behavior is broken or undesirable, > by adding a new member to struct rdns_proposal recording the relative > ordering of proposals within the incoming routing message (i.e. their index > in the proposal set), and using that member in the comparator to sort > addresses of equal priority before resorting to strcmp on the address > string. This presumes the DHCP advertisement order is preserved in the > routing message. There other potential nuances, like multiple proposal sets > (from different routing messages) with the same priority, in which case a > message counter or other mechanism might also be needed to get a more > intuitive total ordering. Naw the qsort() is only sensitive to the address to reduce bigO of the duplicate elimination. I think a slightly more clever sort+eliminate chunk of code would do the job.
Re: hw.perfpolicy behavior on desktop/server
f.holop wrote: > Stuart Henderson - Mon, 09 May 2022 at 17:17:57 > > Currently, you can either set it manually to low speed > > (hw.perfpolicy=manual, hw.setperf=0), modify the kernel (e.g. with the > > diff below), or use obsdfreqd from packages. The latter is only in > > -current packages not 7.1, but it could be built from ports. > > I think the elephant in the room is: > will this change be reverted? > > What is the rationale of not letting wall powered servers > throttle down? As it is today the scheduler-based algorithm seriously sucks, and after the change it was discovered many machines were running 10-20% less than peak performance even under load. This was discovered during suspend/hibernate/resume events but it affects all workloads. No, the change won't get reverted. That is simply trying to ignore the problem with a "I only care about myself" attitude. It may get fixed after work in the scheduler which is not so damned pessimistic to give people 20% less machine than they bought, and this may happen as a resultt of pivoting to the opposite side of the problem space, so that the people who actually performance adjustments ensure the ramping up/down actually works without hurting PERFORMANCE.
Re: hw.perfpolicy behavior on desktop/server
Stuart Henderson - Mon, 09 May 2022 at 17:17:57 > Currently, you can either set it manually to low speed > (hw.perfpolicy=manual, hw.setperf=0), modify the kernel (e.g. with the > diff below), or use obsdfreqd from packages. The latter is only in > -current packages not 7.1, but it could be built from ports. I think the elephant in the room is: will this change be reverted? What is the rationale of not letting wall powered servers throttle down? -f --
Re: HP T430 "Thin Client": Won't sysupgrade without HDMI monitor attached.
Tobias Fiebig - Sat, 07 May 2022 at 13:18:45 > I also very faintly remember people complaining about similar issues > with Mac hardware some years ago (non OpenBSD related, though). The > solution for them back then was soldering together a small plug with > some resistors to make the board 'think' a monitor was connected; also > sold as a commercial product: > https://www.bol.com/nl/nl/p/hdmi-display-port-dummy-plug-4-k-display-emulator/920090918849/ years ago, my intel NUCs also did not boot in eufi mode without hdmi display attached when upgrading. i was just about to buy one of these dummy plugs, but legacy BIOS mode solved it. -f --
Re: calling all PFsync users for experience, gotchas, feedback, tips and tricks
Hi Tom On 5/11/22 21:32, Tom Smyth wrote: We are updating some course material for an upcoming PF firewall course, and I would like to put a call out to those who use PFsync in a redundant firewall cluster The one thing that immediately comes to mind is to NOT use a crossover cable for the pfsync connection (even though that seems to be kind of recommended in the pfsync(4) man page). Doing so will lead to a change of the other firewall's carp demotion counter on its pfsync interface if one peer is rebooted or shut down (and thus causing a link down event on the cabled interface on the other side). It also gives you three chained single points of failure at the same time (nic1, cable, nic2), which I would rather avoid (do the math). I do of course agree with the intention of the suggestion (only run pfsync over a secure link). Since I am in the position where I only run my PF firewalls in a trusted environment, where I also control the switches (no shared cloud etc. infrastructure), I have found that running pfsync over a dedicated VLAN interface on a pair of trunk(4)ed NICs on 2 trusted switches sufficiently satisfies that requirement. Best, Markus
Re: why does resolvd sort nameserver rules
On Wed, May 11, 2022 at 04:54:02PM +0100, james palmer wrote: > i have a local dhcp server running which gives out three nameservers: > > - 192.168.0.2 (resolves some local machine names) > - 9.9.9.9 > - 149.112.112.112 > > on linux, android, and windows the local nameserver takes priority over the > others. > on openbsd thanks to the nameservers being sorted by ip address > 149.112.112.112 is chosen first. > this causes the local machine names to not resolve. > > is there a reason for this behaviour? > i would expect the nameservers to be written in the order they are set as on > the dhcp server. I have no direct knowledge, but judging from the code in resolvd.c handle_route_message and cmp, the sorting is primarily driven by declared priority of the source--resolvers associated with a higher priority source come first, independent of IP address. However, the relative ordering information among a set of proposed resolvers from a particular routing message seems to be dropped on the floor, no other information is used to sort, and the sort comparator falls back to comparing the address strings (using strcmp) to achieve a stable sort. This could be fixed, assuming the current behavior is broken or undesirable, by adding a new member to struct rdns_proposal recording the relative ordering of proposals within the incoming routing message (i.e. their index in the proposal set), and using that member in the comparator to sort addresses of equal priority before resorting to strcmp on the address string. This presumes the DHCP advertisement order is preserved in the routing message. There other potential nuances, like multiple proposal sets (from different routing messages) with the same priority, in which case a message counter or other mechanism might also be needed to get a more intuitive total ordering.
Re: calling all PFsync users for experience, gotchas, feedback, tips and tricks
On 5/11/22 3:32 PM, Tom Smyth wrote: Hello Folks, We are updating some course material for an upcoming PF firewall course, and I would like to put a call out to those who use PFsync in a redundant firewall cluster about your user experience, have you come across any edge cases? have you any tips or tricks about PFSync. have you come across any edge cases / minor misconfigurations / suboptimal configurations that caused problems, were there some tweaks you had to make to make your system scale ? it is likely that people who are running PFSync have more complicated firewall configs. and I would like to see what tuning other people have done in the field. It's been a few years since I managed a firewall cluster with pfsync, but one thing I came up with fairly early on is we needed a way to manage rule changes between the two devices, and I came up with something that I think is pretty cool, and yet haven't seen anyone else describe something similar. Wrote a little script which, when run: * Compared this script on "this" FW with "Other" FW * generated a diff between the /etc/pf.conf file on both systems (other box assumed to be "old", "this" box assumed to be new) * Put the diff into a file along with the user ID of the administrator who made the changes, prompted the user to enter a description for the change above the diff, who approved it, etc. * If the administrator enters a change log and saves the file: * save that file to disk, with a clear date and time stamp. * Copy "this" FW's pf.conf file to the "other" system * pfctl -f /etc/pf.conf on both systems * scp the change log file to the other system * Probably should look for changes in hostname.*, and deal with their changes, too, but I didn't implement that at the time, so I'd be lying if I told you I did. But I recall wishing I had! :D This way, you have a log of every change made to the system, plus administrator comments as to why the change was made. EITHER FW can push changes to the other, both boxes have a full history, either box can be used to rebuild the other. IF you find a problem, a diff to undo it is easily found. It makes change control almost a pleasure. If someone made a change and forgets to push it to the other, you can see that the diff is more complicated than you expected (or you made a typo and blew something out!). I've used similar scripts for other fully redundant systems, like DNS servers. Yes, I'm sure you can do similar things with system management applications like puppet, etc., but this is completely self-contained, no extra hw or packages required. (and yes, DNS has the master/slave config with zone transfers, but I'd argue this is a better system.) Nick.
calling all PFsync users for experience, gotchas, feedback, tips and tricks
Hello Folks, We are updating some course material for an upcoming PF firewall course, and I would like to put a call out to those who use PFsync in a redundant firewall cluster about your user experience, have you come across any edge cases? have you any tips or tricks about PFSync. have you come across any edge cases / minor misconfigurations / suboptimal configurations that caused problems, were there some tweaks you had to make to make your system scale ? it is likely that people who are running PFSync have more complicated firewall configs. and I would like to see what tuning other people have done in the field. I would appreciate any feedback or problem descriptions (with our without solutions) what is the largest throughput firewall you deployed with PFSync? how was your experience of running with PFsync vs without PFsync on your firewall. Thanks again, -- Kindest regards, Tom Smyth.
Re: OpenBSD ports require xbase set - still true?
On 5/9/22 4:56 PM, Steffen Nurpmeso wrote: Hello. Just a rant, not for ports@. I am installing OpenBSD 7.1 right now; this is only a VM, and i want to create / manage ports there. Until now whenever i wanted to do this i had to install xbase, otherwise the port makefile complained some. (I am afraid i have forgotten the details.) Is this still true? So you want to "create/manage" ports in an unsupported environment. What is the "problem" you are trying to solve? This is 2022. Hard drives are measured in hundreds of gigabytes for tiny drives. Current amd64 snapshot, ungzip'd: 21.9M bsd* 22.0M bsd.mp* 4.4M bsd.rd* 592.0M May 11 15:03 base71.tar 265.0M May 11 15:03 comp71.tar 6.3M May 11 15:03 game71.tar 30.5M May 11 15:03 man71.tar 176.0M May 11 15:03 xbase71.tar <-- Not a big deal 35.0M May 11 15:03 xfont71.tar 57.5M May 11 15:03 xserv71.tar 26.6M May 11 15:03 xshare71.tar I think you have a problem with perspective here. All of X (not just xbase) is about 300MB, and just isn't worth worrying about today. What you save by skipping it, you will more than make up for by trying to fix the problems you will make for yourself. Nick.
Re: OpenBSD ports require xbase set - still true?
> ... just a rant ... This is not the list you are looking for. Move along, please. https://www.youtube.com/watch?v=2BmhHyyzh9o
Re: OpenBSD ports require xbase set - still true?
Theo de Raadt wrote in
<74991.1652133...@cvs.openbsd.org>:
|I looked very closely, it started like this
|
| "Just a rant"
|
|And I knew the email was coming from a self-centered individual who is
|unhappy with the entirely volunteer work done by others, yet not unhappy
|enough to quit OpenBSD and switch to another operating sytem where
|there will be similar unhappiness because those other systems also won't
|do precisely what you want.
You could have adjusted this a bit after pasting it. ..mumble..
|Your email is not appropriate. If you don't like OpenBSD, use something
Yes that is true, i really do apologise for the yelling.
(I do recognize that some people can make a living from OpenBSD.)
|else, because noone deserves an email which starts with those 3 words
|you chose, and the following complaint is such horseshit in a world
|where disk drives are cheap. I started OpenBSD a quarter of a century
Oh i have 5GB download per 28 days, with max of ~2.7 MB/sec, but
often less. The next two weeks practically nothing now, there are
so many wars, you do not know where to look first.
|ago by spending $3500 for a 300MB drive and ate noodles and tuna for
By then i also ate tuna. I apologise very, very much. Even our
Russian grocery here had to increase prices by 50 percent now,
peas 1,79€/800G, buckwheat 2,89€/800G, and they use a new recipe
for the ice! That is a real shame. Rice and peas i mean, add
fish sauce and a Vietcong would have been your life long friend.
The times they are a-ch-ch-changing.
The Italiens at least kill the tuna by hand, once a year, in that
terrible slaughter session, "Mattanza". That is minimal respect
compared to artificial robotic slaughter experience. Maybe.
|many months to make up for that, and we do not live in a world where you
|get to moan about 55MB, relative to whatever it takes to ease the compli\
|cate
|work undertaken by the ports developers.
I do not even know whether it is still true, i could not find the
"not installed" i remembered when grepping infrastructure/.
It expands to quite a bit more, the OpenBSD VM ended up as a 1.1GB
image (kernel relink objects removed).
It actually came out like 2.5GB first (!; no games etc.), so
i created a second disk and copied over, leaving off the relink
package. This was quite an experience, i thought about sending
a patch to tech@ that shows up the sequence
fdisk -iy /dev/rsd1c
disklabel -F xx -E /dev/sd1c
newfs /dev/rsd1c
installboot -r XROOT /dev/rsd1a
...copy...
in one of the manual pages of the mentioned programs, i had to
read FAQ on the web to get this together.
Especially "installboot -r", when called from the new volume
mounted to /mnt, did not do what i thought or could deduce from my
(fast, superficial) glance at the manual, it complained about
a readlink failure if i recall correctly (maybe empty path;
unfortunately my tmux history of the ncurses qemu output is
sketchy, i think some terminal sequences cause clearance at times,
for example i have
starting RPC daemons:.
savecore: no core dump
checking quotas: done.
clearing /tmp
So that from boot still
o-0701# installboot -v -r /dev/sd
This must be later, but without meaning
8025728d100b2db6.a / ffs rw,wxallowed 1 1
And this is a cat of the generated file
Again i am wasting your time only, of course).
|In short, Steffen, you need to shut up.
Yeah. Well i am from Hessen and back when we Germans were not
cloned Americans but Germanen the name of our tribe here was
Chatten, and i am afraid this could ring a bell, ne? I do have
watched interviews with you and i know you are prowd of your
sensors ('wish i had some, but you know the ones sit at the front
and the others in the back, and so one group just had bad luck).
But our neighbours from Baden-Württemberg produced a nice animated
film of Chatten i think twenty years ago[1], if you like sensors
mayby this could be a nice 5 minute fun. (Sufficient download
provided.) Warning: people named "Tucker" should be careful.
"Hessi James"[1] of Badesalz, voila.
[1] https://www.youtube.com/watch?v=akfz5Fw-pZI&vl=de
|Steffen Nurpmeso wrote:
Scary top posting everywhere. At least on OpenBSD people seem to
be free enough to no only inject lots of exhaust in toposphere or
where those fly, but also can freely choose their email client and
use plain-text only, which on other BSDs seems to get rare, maybe
$DAYJOB imposed, of course. All those nice HTML mails.
Ciao.
--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter he cheerfully and one by one
|einen nach dem anderen runter wa.ks himself off
|(By Robert Gernhardt)
