Re: futex(2) based pthread_rwlock*
On Thu, Feb 07, 2019 at 06:37:35PM +, Stuart Henderson wrote: > On 2019/01/30 12:29, Martin Pieuchot wrote: > > On 28/01/19(Mon) 14:57, Martin Pieuchot wrote: > > > Summer is really warm here. No need to make my machines hotter by > > > spinning a lot. So here's a futex(2) based pthread_rwlock* > > > implementation. I should look familiar to those of you that reviewed > > > the mutex & semaphore implementations. It uses amotic cas & inc/dec. > > > > > > I moved the existing implementation to rthread_rwlock_compat.c to match > > > what has been done for semaphores. > > > > > > Tests, benchmarks and reviews are more than welcome! > > > > Newer version with fewer atomics and some barrier corrections as pointed > > out by visa@. > > This has been through 3 or 4 bulk ports builds on i386 (2 and 4 core > machines) and running on my amd64 workstation (4 core), no problems seen > on either. Has anyone else been running this? The reduction in spinning > is nice. Yes, I've been running this diff since it was posted on my main laptop (x280). No issues seen and it feels a lot better, especially noticeable when switching back to an install without the diff.
Re: [wou...@nlnetlabs.nl: [NLnet Labs Maintainers] Unbound 1.9.0rc1 pre-release]
On 2019/01/30 22:55, Stuart Henderson wrote: > Diff against /usr/src/usr.sbin/unbound if anyone is interested in testing > the release candidate (there have been a couple of small fixes since). > Release due approximately Friday. 1.9.0 is out, the only change since the rc1 diff is a spelling fix in example.conf.in (which we don't install anyway) so I won't bother to re-attach, https://junkpile.org/unbound-1.9.0.diff if anyone wants the full thing. Any tests / OKs? > This release contains the DNS Flag Day changes for Unbound. See the > reference here, https://dnsflagday.net/ . Or this presentation: > https://indico.dns-oarc.net/event/29/contributions/662/attachments/634/1063/EDNS_Flag_Day_-_OARC29.pdf > . The EDNS timeouts are not used to fallback to nonEDNS queries. > > Out of order processing is implemented, for TCP and TLS. It can be > configured with a maximum amount of memory to use to store pending > answers, and the current memory usage is in the statistics output. This > is with stream-wait-size in unbound.conf and mem.streamwait in > unbound-control stats output. Streams that cause the total memory > counted to exceed the maximum are dropped, but it is possible to get a > number of responses with little memory used. > > There is also TLS session resumption support, that can be enabled with > the tls-session-ticket-keys option. Together with the already existing > TCP fast open, enabled with --enable-tfo-server --enable-tfo-client, > that enables zero RTT stream reconnections to the server. Make sure to > also increase incoming-num-tcp if you expect a lot of TCP and TLS users. > > Options are added to set the TLS ciphers and TLS ciphersuites from > unbound.conf. This can be done with the tls-chiphers and > tls-ciphersuites options. > > TLS can be used from libunbound, with the ub_ctx_set_tls config call, > use that together with ub_ctx_set_fwd to select DNS over TLS transport. > > > Features > - log-tag-queryreply: yes in unbound.conf tags the log-queries and > log-replies in the log file for easier log filter maintenance. > - ip-ratelimit-factor of 1 allows all traffic through, instead of the > previous blocking everything. > - Fix #4206: support openssl 1.0.2 for TLS hostname verification, > alongside the 1.1.0 and later support that is already there. > - Add contrib/unbound-fuzzme.patch from Jacob Hoffman-Andrews, > the patch adds a program used for fuzzing. > - streamtcp option -a send queries consecutively and prints answers > as they arrive. > - out-of-order processing for TCP and TLS. > - Add stream-wait-size: 4m config option to limit the maximum > memory used by waiting tcp and tls stream replies. This avoids > a denial of service where these replies use up all of the memory. > - unbound-control stats has mem.streamwait that counts TCP and TLS > waiting result buffers. > - Patch from Manabu Sonoda with tls-ciphers and tls-ciphersuites > options for unbound.conf. > - Patch for TLS session resumption from Manabu Sonoda, > enable with tls-session-ticket-keys in unbound.conf. > - ub_ctx_set_tls call for libunbound that enables DoT for the machines > set with ub_ctx_set_fwd. Patch from Florian Obser. > > Bug Fixes > - Fix that unbound-checkconf does not complains if the config file > is not placed inside the chroot. > - Refuse to start with no ports. > - Remove clang analysis warnings. > - Patch for typo in unbound.conf man page. > - Fix icon, no ragged edges and nicer resolutions available, for eg. > Win 7 and Windows 10 display. > - cache-max-ttl also defines upperbound of initial TTL in response. > - Fix config parser memory leaks. > - Fix for FreeBSD port make with dnscrypt and dnstap enabled. > - Fixup openssl 1.0.2 compile > - Fix for crash in dns64 module if response is null. > - On FreeBSD warn if systcl settings do not allow server TCP FASTOPEN, > and server tcp fastopen is enabled at compile time. > - Document interaction between the tls-upstream option in the server > section and forward-tls-upstream option in the forward-zone sections. > - Fix syntax in comment of local alias processing. > - Fix NSEC3 record that is returned in wildcard replies from > auth-zone zones with NSEC3 and wildcards. > - Log query name for looping module errors. > - For caps-for-id fallback, use the whitelist to avoid timeout > starting a fallback sequence for it. > - increase mesh max activation count for capsforid long fetches. > - Fix for #4219: secondaries not updated after serial change, unbound > falls back to AXFR after IXFR gives several timeout failures. > - Fix that auth zone after IXFR fallback tries the same master. > - Fix for IXFR fallback to reset counter when IXFR does not timeout. > - Newer aclocal and libtoolize used for generating configure scripts, > aclocal 1.16.1 and libtoolize 2.4.6. > - Fix unit test for python 3.7 new keyword 'async'. > - clang analysis fixes, assert arc4random buffer in init, > no check for already check
Re: futex(2) based pthread_rwlock*
On 2019/01/30 12:29, Martin Pieuchot wrote:
> On 28/01/19(Mon) 14:57, Martin Pieuchot wrote:
> > Summer is really warm here. No need to make my machines hotter by
> > spinning a lot. So here's a futex(2) based pthread_rwlock*
> > implementation. I should look familiar to those of you that reviewed
> > the mutex & semaphore implementations. It uses amotic cas & inc/dec.
> >
> > I moved the existing implementation to rthread_rwlock_compat.c to match
> > what has been done for semaphores.
> >
> > Tests, benchmarks and reviews are more than welcome!
>
> Newer version with fewer atomics and some barrier corrections as pointed
> out by visa@.
This has been through 3 or 4 bulk ports builds on i386 (2 and 4 core
machines) and running on my amd64 workstation (4 core), no problems seen
on either. Has anyone else been running this? The reduction in spinning
is nice.
> Index: libc/include/thread_private.h
> ===
> RCS file: /cvs/src/lib/libc/include/thread_private.h,v
> retrieving revision 1.34
> diff -u -p -r1.34 thread_private.h
> --- libc/include/thread_private.h 10 Jan 2019 18:45:33 - 1.34
> +++ libc/include/thread_private.h 30 Jan 2019 14:21:16 -
> @@ -298,6 +298,10 @@ struct pthread_cond {
> struct pthread_mutex *mutex;
> };
>
> +struct pthread_rwlock {
> + volatile unsigned int value;
> +};
> +
> #else
>
> struct pthread_mutex {
> @@ -314,6 +318,13 @@ struct pthread_cond {
> struct pthread_queue waiters;
> struct pthread_mutex *mutex;
> clockid_t clock;
> +};
> +
> +struct pthread_rwlock {
> + _atomic_lock_t lock;
> + pthread_t owner;
> + struct pthread_queue writers;
> + int readers;
> };
> #endif /* FUTEX */
>
> Index: librthread/Makefile
> ===
> RCS file: /cvs/src/lib/librthread/Makefile,v
> retrieving revision 1.54
> diff -u -p -r1.54 Makefile
> --- librthread/Makefile 12 Jan 2019 00:16:03 - 1.54
> +++ librthread/Makefile 23 Jan 2019 20:07:29 -
> @@ -27,7 +27,6 @@ SRCS= rthread.c \
> rthread_mutex_prio.c \
> rthread_mutexattr.c \
> rthread_np.c \
> - rthread_rwlock.c \
> rthread_rwlockattr.c \
> rthread_sched.c \
> rthread_stack.c \
> @@ -40,9 +39,12 @@ SRCS= rthread.c \
> ${MACHINE_ARCH} == "i386" || ${MACHINE_ARCH} == "mips64" || \
> ${MACHINE_ARCH} == "mips64el" || ${MACHINE_ARCH} == "powerpc" || \
> ${MACHINE_ARCH} == "sparc64"
> -SRCS+= rthread_sem.c
> +CFLAGS+= -DFUTEX
> +SRCS+= rthread_sem.c \
> + rthread_rwlock.c
> .else
> -SRCS+= rthread_sem_compat.c
> +SRCS+= rthread_sem_compat.c \
> + rthread_rwlock_compat.c
> .endif
>
> SRCDIR= ${.CURDIR}/../libpthread
> Index: librthread/rthread.h
> ===
> RCS file: /cvs/src/lib/librthread/rthread.h,v
> retrieving revision 1.63
> diff -u -p -r1.63 rthread.h
> --- librthread/rthread.h 5 Sep 2017 02:40:54 - 1.63
> +++ librthread/rthread.h 23 Jan 2019 17:02:51 -
> @@ -52,13 +52,6 @@ struct stack {
> #define PTHREAD_MAX_PRIORITY31
>
>
> -struct pthread_rwlock {
> - _atomic_lock_t lock;
> - pthread_t owner;
> - struct pthread_queue writers;
> - int readers;
> -};
> -
> struct pthread_rwlockattr {
> int pshared;
> };
> Index: librthread/rthread_rwlock.c
> ===
> RCS file: /cvs/src/lib/librthread/rthread_rwlock.c,v
> retrieving revision 1.11
> diff -u -p -r1.11 rthread_rwlock.c
> --- librthread/rthread_rwlock.c 24 Apr 2018 16:28:42 - 1.11
> +++ librthread/rthread_rwlock.c 30 Jan 2019 14:28:01 -
> @@ -1,8 +1,7 @@
> /* $OpenBSD: rthread_rwlock.c,v 1.11 2018/04/24 16:28:42 pirofti Exp $ */
> /*
> - * Copyright (c) 2004,2005 Ted Unangst
> + * Copyright (c) 2019 Martin Pieuchot
> * Copyright (c) 2012 Philip Guenther
> - * All Rights Reserved.
> *
> * Permission to use, copy, modify, and distribute this software for any
> * purpose with or without fee is hereby granted, provided that the above
> @@ -16,11 +15,7 @@
> * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
> * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
> */
> -/*
> - * rwlocks
> - */
>
> -#include
> #include
> #include
> #include
> @@ -28,6 +23,20 @@
> #include
>
> #include "rthread.h"
> +#include "synch.h"
> +
> +#define UNLOCKED 0
> +#define MAXREADER0x7ffe
> +#define WRITER 0x7fff
> +#define WAITING 0x8000
> +#define COUNT(v) ((v) & WRITER)
> +
> +#define SPIN_COUNT 128
> +#if defined(__i386__) || defined(__amd64__)
> +#define SPIN_WAIT() asm volatile("pause": : : "memory")
> +#else
> +#define SPIN_WAIT() do { } while (0)
> +#endif
>
>
Re: video(1) and modesetting driver
On Fri, Jan 25, 2019 at 09:09:29AM -0200, Martin Pieuchot wrote:
> On 25/01/19(Fri) 11:16, Raphael Graf wrote:
> > On Wed, Jan 23, 2019 at 12:54:51PM -0200, Martin Pieuchot wrote:
> > > > Index: video.c
> > > > ===
> > > > RCS file: /cvs/xenocara/app/video/video.c,v
> > > > retrieving revision 1.26
> > > > diff -u -p -u -p -r1.26 video.c
> > > > --- video.c 4 Jan 2019 17:45:00 - 1.26
> > > > +++ video.c 16 Jan 2019 11:30:21 -
> > > > @@ -38,6 +38,8 @@
> > > > #include
> > > > #include
> > > >
> > > > +#define V_V4L2_PIX_FMT_YUY2v4l2_fourcc('Y', 'U', 'Y', '2')
> > >
> > > Why do we need this define and not the others? Aren't we missing an
> > > include?
> >
> > Actually, I don't know why this define is missing in videoio.h.
> > This YUY2 format (id:0x32595559) is reported by Xv.
> > The uvideo(4) driver reports YUYV which has a define in videoio.h.
> > (YUY2 and YUYV are equivalent)
>
> I see. I believe you're being too clever her. XvListImageFormats(3)
> returns you a list with an Id in FourCC format and you're using V4L2
> defines to match these IDs.
>
> I'd suggest you define only one encoding "yuy2" or "yuyv" and make the
> matching function, find_enc_by_id() a bit more clever. That would
> explicitly document that Xv and V4L2 use different names for the same
> format. On top of that you'll get rid of your conversion code :)
>
The new diff below solves this yuy2/yuyv problem by defining them both under
the same name 'yuy2'.
The only change to the manpage is now the addition of yv12 to the list of
valid encodings.
The trickiest part is the 'choose_enc' function where encodings are chosen
based on device capabilities.
The following conversions are now possible:
yuy2 -> uyvy
yuy2 -> yv12
uyvy -> yuy2
uyvy -> yv12
As my webcam only provides yuy2, I have used input-files for testing:
$ video -i test.yuy2
$ video -i test.uyvy -e uyvy
$ video -i test.yv12 -e yv12
These examples work for me with both drivers (modesetting and intel).
The conversion to yv12 has a small performance impact, of course. Do you think
the performance is acceptable?
> > In the new diff, I have changed the variable names enc_in/enc_out to
> > enc/enc_xv
> > as the second 'enc' is only used for Xv output.
> > I also fixed the style(9) issues and added a CONV_NONE define.
>
> I find enc_in/enc_out more explicit, but let's keep it like that for
> now. This also makes the diff smaller :)
>
> > Note that I'm not sure if the algorithm used in the yuy2_to_yv12 function is
> > correct in respect to color theory..
> > (It is a conversion from 16 to 12 bits/pixel)
>
> I'm sure you can find some good documentation and/or example about that
> on the Web or in the existing freesoftware ecosystem.
>
Yes, there are libraries like 'libyuv' doing it in the same way.
(they also have optimized versions (SSE etc.))
Any comments, questions or feedback?
Index: video.1
===
RCS file: /cvs/xenocara/app/video/video.1,v
retrieving revision 1.13
diff -u -p -u -p -r1.13 video.1
--- video.1 4 Jun 2016 07:44:32 - 1.13
+++ video.1 7 Feb 2019 12:08:37 -
@@ -84,9 +84,10 @@ The default is 0, the first adaptor repo
.It Fl e Ar encoding
Lowercase FOURCC name of video encoding to use.
Valid arguments are
-.Ql uyvy
+.Ql uyvy ,
+.Ql yuy2
and
-.Ql yuy2 .
+.Ql yv12 .
The default is
.Ql yuy2
unless
Index: video.c
===
RCS file: /cvs/xenocara/app/video/video.c,v
retrieving revision 1.27
diff -u -p -u -p -r1.27 video.c
--- video.c 22 Jan 2019 20:02:40 - 1.27
+++ video.c 7 Feb 2019 12:08:37 -
@@ -38,6 +38,9 @@
#include
#include
+/* Synonym for YUYV */
+#define V_V4L2_PIX_FMT_YUY2v4l2_fourcc('Y', 'U', 'Y', '2')
+
/* Xv(3) adaptor properties */
struct xv_adap {
char name[128];
@@ -133,20 +136,22 @@ struct dev {
/* video encodingss */
struct encodings {
char*name;
+ int id;
int bpp;
- int xv_id;
- int dev_id;
-#defineSW_DEV 0x1
-#defineSW_XV 0x2
-#define SW_MASK(SW_DEV | SW_XV)
+#defineSUP_DEV 0x1
+#defineSUP_XV 0x2
int flags;
} encs[] = {
+#define ENC_YUYV 1
+ { "yuy2", V4L2_PIX_FMT_YUYV, 16, 0 },
#define ENC_YUY2 0
- { "yuy2", 16, -1, -1, 0 },
-#define ENC_UYVY 1
- { "uyvy", 16, -1, -1, 0 },
-#define ENC_LAST 2
- { NULL, 0, 0, 0, 0 }
+ { "yuy2", V_V4L2_PIX_FMT_YUY2, 16, 0 },
+#define ENC_UYVY 2
+ { "uyvy", V4L2_PIX_FMT_UYVY, 16, 0 },
+#define ENC_YV12 3
+ { "yv12", V4L2_PIX_FMT_YVU420, 12, 0 },
+#define ENC_LAST 4
+ { NULL, 0, 0, 0 }
};
struct video {
@@ -162,9 +167,12 @@ struct video {
char iofile[FILENAME_MAX];
int
Re: Switch powerpc to big PIC
> From: Philip Guenther
> Date: Tue, 5 Feb 2019 18:40:09 -0800
>
> On Tue, Feb 5, 2019 at 12:59 PM Mark Kettenis
> wrote:
>
> The architecture already has big PIE. The issue is that clang doesn't
> support secure-plt for small pic. I haven't entirely figured out
> what's going on here and we probably need some further fixes to clang
> here. On the other hand I think it is probably time to recognize
> there is more and more bloat in the world.
>
> Thoughts?
>
> libexec/ld.so/powerpc/Makefile.inc needs updating to match, no?
> ---
> CFLAGS += -fpic -msoft-float
> ---
Ultimately, yes. But my immediate goal is to make it possible to
build libc++abi and libc++.
> Also, does clang support secure-plt for small pie? If not, then
> lib/csu/Makefile needs updating too:
> ---
> # Override powerpc default of -fPIE
> # XXX if this is safe, why not override CFLAGS for alpha and sparc64 too?
> # Does it work because the csu bits come first and get the first few GOT
> # entries?
> .if ${MACHINE_ARCH} == "powerpc"
> CFLAGS+=-fpie
> .endif
I still need to investigate this a bit further. But fi there are no
downsides removing this bit, we probably should at some point.
bgpd L3VPN rework
At a2k19 David (dlg@) and I sat down and looked at BGP L3 MPLS VPN
support. Now David wants to use this in production and realized that
in his case it would be great to have more than one mpe(4) interface per
rdomain. This way it is possible to write good firewall rules using
interface names or groups.
The definition of VPNs in bgpd was never super elegant. The 'depend on
mpeX' config was a bit redundant and so after some discussions we decided
to rework this part.
L3 MPLS VPN are now configured like this:
vpn "staff" on mpe1 {
rd $ASN:1
import-target rt $ASN:100
export-target rt $ASN:101
network 0/0
}
vpn "users" on mpe2 {
rd $ASN:2
import-target rt $ASN:200
export-target rt $ASN:201
network 0/0
}
Now in this example mpe1 and mpe2 can be in the same rdomain. The rdomain
is now selected based on the rdomain of the mpe(4) interface. There are
probably still some gotchas around this which can be tackled in a 2nd
round.
This diff does a lot of shuffling of code and especially affect network
statements (since those are now per VPN and no longer per rdomain).
The rewrite of the network code fixed also some other behaviour bugs which
do not only affect VPN setups. In short conficts between 'network A.B.C.D/N'
and 'network static' are now properly handled (with 'network A.B.C.D/N'
having preference).
Since this is a major change please test (or suffer later).
--
:wq Claudio
Index: usr.sbin/bgpctl/bgpctl.c
===
RCS file: /cvs/src/usr.sbin/bgpctl/bgpctl.c,v
retrieving revision 1.228
diff -u -p -r1.228 bgpctl.c
--- usr.sbin/bgpctl/bgpctl.c20 Jan 2019 23:30:15 - 1.228
+++ usr.sbin/bgpctl/bgpctl.c7 Feb 2019 10:11:19 -
@@ -346,7 +346,7 @@ main(int argc, char *argv[])
bzero(&net, sizeof(net));
net.prefix = res->addr;
net.prefixlen = res->prefixlen;
- net.rtableid = tableid;
+ net.rd = res->rd;
/* attribute sets are not supported */
if (res->action == NETWORK_ADD) {
imsg_compose(ibuf, IMSG_NETWORK_ADD, 0, 0, -1,
@@ -927,7 +927,7 @@ show_fib_head(void)
printf("flags: "
"* = valid, B = BGP, C = Connected, S = Static, D = Dynamic\n");
printf(" "
- "N = BGP Nexthop reachable via this route R = redistributed\n");
+ "N = BGP Nexthop reachable via this route\n");
printf(" r = reject route, b = blackhole route\n\n");
printf("flags prio destination gateway\n");
}
@@ -969,11 +969,6 @@ show_fib_flags(u_int16_t flags)
else
printf(" ");
- if (flags & F_REDISTRIBUTED)
- printf("R");
- else
- printf(" ");
-
if (flags & F_REJECT && flags & F_BLACKHOLE)
printf("f");
else if (flags & F_REJECT)
@@ -1983,7 +1978,7 @@ network_bulk(struct parse_result *res)
errx(1, "bad prefix: %s", b);
net.prefix = h;
net.prefixlen = len;
- net.rtableid = tableid;
+ net.rd = res->rd;
if (res->action == NETWORK_BULK_ADD) {
imsg_compose(ibuf, IMSG_NETWORK_ADD,
@@ -2128,7 +2123,7 @@ network_mrt_dump(struct mrt_rib *mr, str
net.prefix = ctl.prefix;
net.prefixlen = ctl.prefixlen;
net.type = NETWORK_MRTCLONE;
- /* XXX rtableid */
+ /* XXX rd can't be set and will be 0 */
imsg_compose(ibuf, IMSG_NETWORK_ADD, 0, 0, -1,
&net, sizeof(net));
Index: usr.sbin/bgpctl/parser.c
===
RCS file: /cvs/src/usr.sbin/bgpctl/parser.c,v
retrieving revision 1.89
diff -u -p -r1.89 parser.c
--- usr.sbin/bgpctl/parser.c20 Jan 2019 23:30:15 - 1.89
+++ usr.sbin/bgpctl/parser.c7 Feb 2019 10:11:19 -
@@ -58,6 +58,7 @@ enum token_type {
PREPNBR,
PREPSELF,
WEIGHT,
+ RD,
FAMILY,
GETOPT,
RTABLE,
@@ -374,6 +375,11 @@ static const struct token t_network_show
{ ENDTOKEN, "", NONE, NULL}
};
+static const struct token t_rd[] = {
+ { RD, "", NONE, t_set},
+ { ENDTOKEN, "", NONE, NULL}
+};
+
static const struct token t_set[] = {
{ NOTOKEN, "", NONE, NULL},
{ KEYWORD, "community",NONE, t_community},
@@ -386,6 +392,7 @@ static const struct token t_set[] = {
{ KEYWORD, "pftable", NONE, t_pftable},
{ KEYWORD, "prepend-neighbor", NONE, t_prepnbr},
{ KEYWORD, "prepend-self", NO
