Re: in which we present an ugly hack to make sys/queue.h CIRCLEQ work

[David Holland]

On Thu, Nov 21, 2013 at 01:16:44AM +0100, Rhialto wrote:
 > Ever since I grokked the elegance of Lists in AmigaOS, I've always
 > wondered why other list implementations do it differently.

One reason is that with Amiga lists is that the list node structure
needs to be at the beginning of the object, and consequently any
particular object can only be on one list. This is ok for exec.library
(and a lot of other stuff) but falls down in many cases. Part of the
supposed advantage of the  types is that they don't have
this limitation.

Also, with current C and C compilers you really can't share the null
end pointer; you need two list nodes in the list head, and they need
to specifically be instances of the list node structure. Having only
the one null not only violates the strict-aliasing rules but also a
bunch of regulations about structure member padding and alignment.

 > This initially confusing situation eliminates all special cases for
 > node insertion and removal (and traversal isn't made more complicated).

Traversal is slightly more complicated, in that the start and end
logic has to skip the bookends, and until you're used to the idiom
it's easy to mess this up. And if you do, demons fly out of your
nose...

However, in general I agree with you, it's much more elegant.

-- 
David A. Holland
dholl...@netbsd.org

Re: in which we present an ugly hack to make sys/queue.h CIRCLEQ work

[David Holland]

On Wed, Nov 20, 2013 at 07:01:15PM -0500, Ken Hornstein wrote:
 > #define TAILQ_PREV(elm, headname, field)\
 > (*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))

There's another wrinkle, however, which is that this code (TAILQ_PREV)
also violates the strict-aliasing rules. I don't think anyone has
found a clear case of gcc (4.8 or otherwise) tripping on it yet, but
it too really ought to be fixed before it bites someone.

-- 
David A. Holland
dholl...@netbsd.org

Re: in which we present an ugly hack to make sys/queue.h CIRCLEQ work

[Ken Hornstein]

>I think it is actually 2 extra pointer deferences,

You're right, my apologies.

>but the important bit
>is that one of the pointers is fetched from memory you might not need to
>read from at all with a CIRCLEQ.  On modern processors one cache miss is
>worth a whole big pile of extra instructions, so doing reads from memory
>you don't strictly need to touch is about the easiest way to make things
>go slower.

I can understand that all, but still ... are people really using this in
ways it would be noticable?  If it was a serious issue, we could add the
extra previous element pointer you're talking about to the TAILQ macros
and it seems like that would do everything you want.

In a more practical sense ... it seems the only users of CIRCLEQ_PREV
are in sys/kern/subr_vmem.c and sys/kern/vfs_mount.c.  In a lot of the
ones in subr_vmem.c look like the relevant pointers would already be in
the cache (and they're only going back one element, not traversing the
whole list). vfs_mount.c uses in vfs_unmountall1().  CIRCLEQ_LAST is
only used by libform and also in vfs_mount.c.

--Ken

Re: in which we present an ugly hack to make sys/queue.h CIRCLEQ work

[Dennis Ferguson]

On 20 Nov, 2013, at 16:01 , Ken Hornstein  wrote:
>> This functionally works, but the TAILQ data structure is not the same as
>> the CIRCLEQ data structure (i.e. no ABI compatibility, if that matters;
>> I'm not quite sure why it does) and unnecessarily penalizes applications
>> which need to traverse the list in the tail->head direction.
> 
> I know the data structures aren't the same, but if you're using the macros
> you don't ever notice this.As for the penalty ... you're talking
> about:
> 
> #define CIRCLEQ_PREV(elm,field) ((elm)->field.cqe_prev)
> 
> versus
> 
> #define TAILQ_PREV(elm, headname, field)\
>(*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))
> 
> One extra pointer dereference is what we're talking about; I have to
> believe that for 99% of applications it wouldn't matter.

I think it is actually 2 extra pointer deferences, but the important bit
is that one of the pointers is fetched from memory you might not need to
read from at all with a CIRCLEQ.  On modern processors one cache miss is
worth a whole big pile of extra instructions, so doing reads from memory
you don't strictly need to touch is about the easiest way to make things
go slower.

> Also, I see that CIRCLEQ has more complicated logic if you want to
> insert elements at the end of it.  So obviously there are tradeoffs (but
> again, I don't really think it matters to nearly everybody).

I don't think an extra instruction or two matters nearly as much as extra
reads from unrelated memory.  I was unable to measure any difference between
the speed of adding something to the end of a TAILQ and adding something to
the end of a CIRCLEQ, but I was able to measure a difference between
TAILQ_LAST() and CIRCLEQ_LAST() for a list that multiple processors were
adding things to.

Dennis Ferguson

Re: in which we present an ugly hack to make sys/queue.h CIRCLEQ work

[Rhialto]

On Wed 20 Nov 2013 at 15:32:20 -0800, Dennis Ferguson wrote:
> If one were starting this from scratch [...]

Ever since I grokked the elegance of Lists in AmigaOS, I've always
wondered why other list implementations do it differently.

A list is double linked, and it has dummy first and last nodes (which
aren't part of the data and consist just of the next and prev pointers).
They are elegantly combined in the list header, which consists of 3
pointers:

mlh_Headwhich points to the first real node,
mlh_Tailwhich is always NULL
mlh_TailPredwhich points to the last real node.

Each node in the list starts with a ln_Next and a ln_Prev pointer.

When the list is empty,

mlh_Headpoints to mlh_Tail
mlh_Tailis always NULL
mlh_TailPredpoints to mlh_Head.

The first dummy node consists of mlh_Head and mlh_Tail, i.e. its ln_Prev
is NULL, and the last dummy node consists of  mlh_Tail and mlh_TailPred,
where its ln_Next is NULL.

This initially confusing situation eliminates all special cases for
node insertion and removal (and traversal isn't made more complicated).

This is more extensively explained in
http://wiki.amigaos.net/index.php/Exec_Lists_and_Queues .

-Olaf.
-- 
___ Olaf 'Rhialto' Seibert  -- The Doctor: No, 'eureka' is Greek for
\X/ rhialto/at/xs4all.nl-- 'this bath is too hot.'



pgpGoRV51t32L.pgp
Description: PGP signature

Re: in which we present an ugly hack to make sys/queue.h CIRCLEQ work

[Ken Hornstein]

>This functionally works, but the TAILQ data structure is not the same as
>the CIRCLEQ data structure (i.e. no ABI compatibility, if that matters;
>I'm not quite sure why it does) and unnecessarily penalizes applications
>which need to traverse the list in the tail->head direction.

I know the data structures aren't the same, but if you're using the macros
you don't ever notice this.As for the penalty ... you're talking
about:

#define CIRCLEQ_PREV(elm,field) ((elm)->field.cqe_prev)

versus

#define TAILQ_PREV(elm, headname, field)\
(*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))

One extra pointer dereference is what we're talking about; I have to
believe that for 99% of applications it wouldn't matter.

Also, I see that CIRCLEQ has more complicated logic if you want to
insert elements at the end of it.  So obviously there are tradeoffs (but
again, I don't really think it matters to nearly everybody).

--Ken

--Ken

Re: in which we present an ugly hack to make sys/queue.h CIRCLEQ work

[Dennis Ferguson]

On 20 Nov, 2013, at 12:32 , Zhihao Yuan  wrote:
> On Wed, Nov 20, 2013 at 12:07 PM, Ken Hornstein  wrote:
>> I see that at least on MacOS X, sys/queue.h contains the following note:
>> 
>> * Note that circle queues are deprecated, because, as the removal log
>> * in FreeBSD states, "CIRCLEQs are a disgrace to everything Knuth taught
>> * us in Volume 1 Chapter 2. [...] Use TAILQ instead, it provides the same
>> * functionality." Code using them will continue to compile, but they
>> * are no longer documented on the man page.
>> 
>> I don't think I'm knowledgable enough to speak to the accuracy of that
>> comment, but would it be simpler just to migrate all of the code over to
>> TAILQ if the CIRCLEQ ABI is broken as designed?
> 
> It should not be too hard.  Actually, after I mirgrated CIRCLEQ to
> TAILQ in nvi2's code,
> 
>  https://github.com/lichray/nvi2/commits/master?page=9
> 
> FreeBSD base already has absolutely no code using CIRCLEQ.
> 
> So my suggestion is simple: nuke it out.

This functionally works, but the TAILQ data structure is not the same as
the CIRCLEQ data structure (i.e. no ABI compatibility, if that matters; I'm
not quite sure why it does) and unnecessarily penalizes applications which
need to traverse the list in the tail->head direction.

If one were starting this from scratch I don't think there would be
separate TAILQ and CIRCLEQ implementations, there would instead be one
implementation which used the CIRCLEQ data structure but with NULL in
the .cqe_prev member of the head of the list and the .cqe_next member
at the tail.  TAILQ users couldn't tell the difference between this and
the current TAILQ structure in terms of either performance or API, save
for the fact that TAILQ_INSERT_BEFORE() would need to acquire a pointer
to head like the other TAILQ_INSERT_*() macros have, while CIRCLEQ users
wouldn't be penalized for wanting a list which works equally well when
moving in either direction but would get to detect the end of list (in
either direction) with a more conventional comparison to NULL rather
than the disgusting thing they have to do now.

Dennis Ferguson

Re: in which we present an ugly hack to make sys/queue.h CIRCLEQ work

[Zhihao Yuan]

On Wed, Nov 20, 2013 at 12:07 PM, Ken Hornstein  wrote:
> I see that at least on MacOS X, sys/queue.h contains the following note:
>
>  * Note that circle queues are deprecated, because, as the removal log
>  * in FreeBSD states, "CIRCLEQs are a disgrace to everything Knuth taught
>  * us in Volume 1 Chapter 2. [...] Use TAILQ instead, it provides the same
>  * functionality." Code using them will continue to compile, but they
>  * are no longer documented on the man page.
>
> I don't think I'm knowledgable enough to speak to the accuracy of that
> comment, but would it be simpler just to migrate all of the code over to
> TAILQ if the CIRCLEQ ABI is broken as designed?

It should not be too hard.  Actually, after I mirgrated CIRCLEQ to
TAILQ in nvi2's code,

  https://github.com/lichray/nvi2/commits/master?page=9

FreeBSD base already has absolutely no code using CIRCLEQ.

So my suggestion is simple: nuke it out.

-- 
Zhihao Yuan, ID lichray
The best way to predict the future is to invent it.
___
4BSD -- http://4bsd.biz/

Re: in which we present an ugly hack to make sys/queue.h CIRCLEQ work

[Dennis Ferguson]

On 20 Nov, 2013, at 01:54 , matthew green  wrote:
> unfortunately, changing these macros to be strictly C requires changing
> the ABI of them, and while we are going to consider such a change, in
> the interest of getting things working we present the following hack.
> it was inspired by David Holland, and we considered placing it in
> sys/cdefs.h for other consumers, but as queue.h currently only relies
> on the presence of "NULL" combined with the need for ""
> to work correctly for the tools build (which may be on non-netbsd
> platforms.)

I may be misunderstanding something, but it seems like this might
preserve the ABI (e.g. old binaries with a newly-compiled library)
but still represents a change to the API for anything newly
compiled(?).  That is, doesn't this code from the kernel have the
same problem that the macros themselves have, or am I missing something?

/* 
 * sigclear:
 * 
 *  Clear all pending signals in the specified set. 
 */
void  
sigclear(sigpend_t *sp, const sigset_t *mask, ksiginfoq_t *kq)
{ 
ksiginfo_t *ksi, *next; 
 
if (mask == NULL)
sigemptyset(&sp->sp_set);
else 
sigminusset(mask, &sp->sp_set);
 
ksi = CIRCLEQ_FIRST(&sp->sp_info); 
for (; ksi != (void *)&sp->sp_info; ksi = next) {
next = CIRCLEQ_NEXT(ksi, ksi_list);
if (mask == NULL || sigismember(mask, ksi->ksi_signo)) {
CIRCLEQ_REMOVE(&sp->sp_info, ksi, ksi_list);
KASSERT((ksi->ksi_flags & KSI_FROMPOOL) != 0);
KASSERT((ksi->ksi_flags & KSI_QUEUED) != 0);
CIRCLEQ_INSERT_TAIL(kq, ksi, ksi_list);
}
}
}

So is the plan to add the ugly inline to the CIRCLEQ_* macro
definitions, and then fix each program which uses the macros
with the same ugly inline when they are compiled by the new
compiler?

If that's the plan then it seems like it might be better to know
what the final solution looks like before changing any code which
uses the CIRCLEQ macros, so that any code which needs to be fixed
just needs to be fixed once.

Dennis Ferguson

Re: in which we present an ugly hack to make sys/queue.h CIRCLEQ work

[Daode]

matthew green  wrote:
 |while preparing to update to GCC 4.8 i discovered that our sys/queue.h
 |CIRCLEQ macros violate C aliasing rules, ultimately leading to the
 |compiler eliding comparisons it declared as always false.

I am far away from penetrating these C sequence point and aliasing
semantics, and face it in an unacademic way on a case-by-case base.
But couldn't this particular problem be truly solved by enwrapping
all of it in unions, so that the tests end up as tests against
char* or integer:

  -#define  CIRCLEQ_EMPTY(head) ((head)->cqh_first == (void 
*)(head))
  +#define  CIRCLEQ_EMPTY(head) ((head)->_d.cqh_first._i == 
(uintptr_t)&(head)->_x[0])

A blind dash off with _INIT(), _INSERT_HEAD(), _FOREACH() and
_EMPTY().  Maybe the inner unions are redundant, so; i have never
used queue.3 and thus didn't know what i really need, but.
Even if, it'd need stdint.h for uintptr_t (or stddef.h for
ptrdiff_t).

In fact i'm not really convinced from
  + for ((var) = ((head)->_d.cqh_first._p); \
  + (uintptr_t)(var) != (uintptr_t)&(head)->_x[0];  \
  + (var) = ((var)->field._d.cqe_next._p))
at the moment.  (hmm.)

  --- queue.h.orig  2013-11-20 20:43:01.0 +0100
  +++ queue.h   2013-11-20 20:55:33.0 +0100
  @@ -640,27 +640,46 @@ struct {
\
   #define QUEUEDEBUG_CIRCLEQ_POSTREMOVE(elm, field)
   #endif
   
  +#include 
   #define  CIRCLEQ_HEAD(name, type)
\
  -struct name {
\
  - struct type *cqh_first; /* first element */ \
  - struct type *cqh_last;  /* last element */  \
  +union name { \
  + struct {\
  + union { \
  + uintptr_t   _i; \
  + struct type *_p;/* first element */ \
  + }   cqh_first;  \
  + union { \
  + uintptr_t   _i; \
  + struct type *_p;/* first element */ \
  + }   cqh_last;   /* last element */  \
  + }   _d; \
  + char_x[2 * sizeof(uintptr_t)];  \
   }
   
   #define  CIRCLEQ_HEAD_INITIALIZER(head)  
\
  - { (void *)&head, (void *)&head }
  + { {{(uintptr_t)&head}, {(uintptr_t)&head}} }
   
   #define  CIRCLEQ_ENTRY(type) 
\
  -struct { \
  - struct type *cqe_next;  /* next element */  \
  - struct type *cqe_prev;  /* previous element */  \
  +union {  
\
  + struct {\
  + union { \
  + uintptr_t   _i; \
  + struct type *_p;\
  + }   cqe_next;   /* next element */  \
  + union { \
  + uintptr_t   _i; \
  + struct type *_p;\
  + }   cqe_prev;   /* previous element */  \
  + }   _d; \
  + char_x[2 * sizeof(uintptr_t)];  \
   }
   
   /*
* Circular queue functions.
*/
   #define  CIRCLEQ_INIT(head) do { 
\
  - (head)->cqh_first = (void *)(head); \
  - (head)->cqh_last = (void *)(head);  \
  + (head)->_d.cqh_first._i = (uintptr_t)(head);\
  + (head)->_d.cqh_last._i = (uintptr_t)(head); \
   } while (/*CONSTCOND*/0)
   
   #define  CIRCLEQ_INSERT_AFTER(head, listelm, elm, field) do {
\
  @@ -689,13 +708,13 @@ struct {
\
   
   #define  CIRCLEQ_INSERT_HEAD(head, elm, field) do {  
\
QUEUEDEBUG_CIRCLEQ_HEAD((head), field)  \
  - (elm)->field.cqe_next = (head)->cqh_first;  \
  -

Re: A Library for Converting Data to and from C Structs for Lua

[Marc Balmer]

Am 18.11.13 00:46, schrieb John Nemeth:
> On Nov 17, 11:02pm, Marc Balmer wrote:
> } Am 17.11.13 20:40, schrieb Lourival Vieira Neto:
> } > On Sun, Nov 17, 2013 at 4:39 PM, David Holland  
> wrote:
> } >> On Sun, Nov 17, 2013 at 01:32:03PM +0100, Hubert Feyrer wrote:
> } >>  > >I plan to import it and to make it available to both lua(1) and 
> lua(4)
> } >>  >
> } >>  > I wonder if we really need to get all this into NetBSD,
> } >>  > instead of moving it to pkgsrc somehow.
> } >>
> } >> This...
> } > 
> } > I think that would be nice to have Lua kernel modules in pkgsrc, if 
> possible.
> } 
> } No, I don't think so.  They interact to much with the system, they need
> } to be part of the system.
> 
>  Uh, no.  The whole idea behind modules clearly means being
> able to use third party code.  We should be able to have modules
> in pkgsrc.  There are no modules in pkgsrc yet, but that's just a
> matter of figuring out the best way to do it.  There is no reason
> why all modules must be included with the system.

The code we write, in contrast to maybe other code, is not third party
code but code written by NetBSD developers for use in NetBSD only.  It
has no business in pkgsrc and pkgsrc does not provide us with the
infrastructure needed to build and maintain the code.  So we happily
stick with the NetBSD Makefile fragments infrastructure which suits our
needs perfectly.

Re: A Library for Converting Data to and from C Structs for Lua

[Justin Cormack]

On 20 Nov 2013 08:38, "Marc Balmer"  wrote:
> > Now we need a name that covers both uses cases.  It could be "memory"
> > because it deals with memory, or just "data", which I favour.
> >
> > Opinions on the name?
>
> Since no one replied, it will go by the name 'data' and be available for
> "both" Luas.
>
> @lneto: I will start with the pack/unpack parts, you can then add your
> stuff whenever you want, ok?

I don't have opinions on the name but I do have a set of feature
requirements. I am currently using luaffi which needs some work to remove
the non portable parts, but that's not far off. Happy to switch but I do
need access to struct members like tables, nested structs, unions, casts,
metatables for structs. If there was an outline design doc that would be
helpful.

Re: in which we present an ugly hack to make sys/queue.h CIRCLEQ work

[Matthew Orgass]

On 2013-11-20 m...@eterna.com.au wrote:


i'm going to commit this soon, but if anyone has more useful hacks or
real fixes, please let me/these lists know and we'll consider them.



+ * If we ever choose to change the ABI of the CIRCLEQ macros, we could fix
+ * this by changing the


  If changing the ABI (or for places where it doesn't matter), please 
consider gcq (in gcq.h, gcq(3)).  Or use part of it (I think the *_TYPED 
macros are drop in replacements for CIRCLEQ).  Also, slhchi is the only 
thing in the tree that uses it and while in theory some third party 
software could use it, I'd assume that in practice anyone who did would 
just copy it, so it wouldn't be a problem to alter the interface some if 
needed.  Also, while I did test it at the time I haven't used it again yet 
so it is possible there are bugs lurking outside of what slhci uses.


  I don't fully understand the strict aliasing rules, but the only thing 
that seems like it might cause trouble is gcq_head(), which is not used 
internally but provided to be able to treat any element as the head for 
iteration purposes (presumably because you don't have a designated head, 
so not CIRCLEQ usage).  It casts a struct to a struct that contains only 
that struct and is used only to cast back.  If that isn't safe, that 
function could just be removed and replaced with an explanation of why it 
isn't safe, however I expect it to be safe because it is never used except 
to cast to the inner type (the head only has a separate type to make it 
hard to accidently mess up the argument order).


-Matt

Re: in which we present an ugly hack to make sys/queue.h CIRCLEQ work

[Ken Hornstein]

>while preparing to update to GCC 4.8 i discovered that our sys/queue.h
>CIRCLEQ macros violate C aliasing rules, ultimately leading to the
>compiler eliding comparisons it declared as always false.

I see that at least on MacOS X, sys/queue.h contains the following note:

 * Note that circle queues are deprecated, because, as the removal log
 * in FreeBSD states, "CIRCLEQs are a disgrace to everything Knuth taught
 * us in Volume 1 Chapter 2. [...] Use TAILQ instead, it provides the same
 * functionality." Code using them will continue to compile, but they
 * are no longer documented on the man page.

I don't think I'm knowledgable enough to speak to the accuracy of that
comment, but would it be simpler just to migrate all of the code over to
TAILQ if the CIRCLEQ ABI is broken as designed?

--Ken

Re: Replace VI_INACT*

[David Holland]

On Tue, Nov 19, 2013 at 11:57:14AM +0100, J. Hannken-Illjes wrote:
 > The attached diff will:
 > 
 > Replace VI_INACTNOW and VI_INACTREDO with a new flag VI_CHANGING that gets
 > set while a vnode changes state from active to inactive or from active
 > or inactive to clean and protects "vclean(); vrelel()" and "vrelel()"
 > against "vget()".
 > 
 > Comments or objections anyone?

I don't see any obvious problems and it looks like a step forward...

-- 
David A. Holland
dholl...@netbsd.org

Re: A Library for Converting Data to and from C Structs for Lua

[Lourival Vieira Neto]

On Wed, Nov 20, 2013 at 10:38 AM, Marc Balmer  wrote:
> Am 20.11.13 13:33, schrieb Lourival Vieira Neto:
>> On Wed, Nov 20, 2013 at 6:36 AM, Marc Balmer  wrote:
>>> Am 18.11.13 09:07, schrieb Marc Balmer:
 Am 17.11.13 13:05, schrieb Marc Balmer:
> I came accross a small library for converting data to an from C structs
> for Lua, written by Roberto Ierusalimschy:
>
> http://www.inf.puc-rio.br/~roberto/struct/
>
> I plan to import it and to make it available to both lua(1) and lua(4)
> as follows:
>
> The source code will be imported into
> ${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
> to compile on NetBSD.
>
> Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
> ${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
> respective Makefiles etc.

 After discussion we lneto@ and others we realised that there are several
 such libraries around, and that I as well as lneto@ wrote one.  SO we
 decided to merge our works and have a library that allows to
 encode/decode structured binary data in one call (my work) and that
 allows to access individual fields as well (lneto@'s work).

 Now we need a name that covers both uses cases.  It could be "memory"
 because it deals with memory, or just "data", which I favour.

 Opinions on the name?
>>>
>>> Since no one replied, it will go by the name 'data' and be available for
>>> "both" Luas.
>>
>> 'Data' is fine for me; I don't have a better suggestion anyway =).
>>
>>> @lneto: I will start with the pack/unpack parts, you can then add your
>>> stuff whenever you want, ok?
>>
>> Just fine for me. However, I think we need to define the API before
>> (re)starting coding.
>
> Yes.  I will meet Justin at LWS, discuss with him, and then mail out
> what we discussed, so you can chime in.  ok?

Alright.. I'll try to send some thoughts too in the meanwhile..

-- 
Lourival Vieira Neto

Re: A Library for Converting Data to and from C Structs for Lua

[Marc Balmer]

Am 20.11.13 13:33, schrieb Lourival Vieira Neto:
> On Wed, Nov 20, 2013 at 6:36 AM, Marc Balmer  wrote:
>> Am 18.11.13 09:07, schrieb Marc Balmer:
>>> Am 17.11.13 13:05, schrieb Marc Balmer:
 I came accross a small library for converting data to an from C structs
 for Lua, written by Roberto Ierusalimschy:

 http://www.inf.puc-rio.br/~roberto/struct/

 I plan to import it and to make it available to both lua(1) and lua(4)
 as follows:

 The source code will be imported into
 ${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
 to compile on NetBSD.

 Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
 ${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
 respective Makefiles etc.
>>>
>>> After discussion we lneto@ and others we realised that there are several
>>> such libraries around, and that I as well as lneto@ wrote one.  SO we
>>> decided to merge our works and have a library that allows to
>>> encode/decode structured binary data in one call (my work) and that
>>> allows to access individual fields as well (lneto@'s work).
>>>
>>> Now we need a name that covers both uses cases.  It could be "memory"
>>> because it deals with memory, or just "data", which I favour.
>>>
>>> Opinions on the name?
>>
>> Since no one replied, it will go by the name 'data' and be available for
>> "both" Luas.
> 
> 'Data' is fine for me; I don't have a better suggestion anyway =).
> 
>> @lneto: I will start with the pack/unpack parts, you can then add your
>> stuff whenever you want, ok?
> 
> Just fine for me. However, I think we need to define the API before
> (re)starting coding.

Yes.  I will meet Justin at LWS, discuss with him, and then mail out
what we discussed, so you can chime in.  ok?

Re: A Library for Converting Data to and from C Structs for Lua

[Lourival Vieira Neto]

On Wed, Nov 20, 2013 at 7:31 AM, Marc Balmer  wrote:
> Am 20.11.13 10:26, schrieb Justin Cormack:
>>
>> On 20 Nov 2013 08:38, "Marc Balmer" mailto:m...@msys.ch>>
>> wrote:
>>> > Now we need a name that covers both uses cases.  It could be "memory"
>>> > because it deals with memory, or just "data", which I favour.
>>> >
>>> > Opinions on the name?
>>>
>>> Since no one replied, it will go by the name 'data' and be available for
>>> "both" Luas.
>>>
>>> @lneto: I will start with the pack/unpack parts, you can then add your
>>> stuff whenever you want, ok?
>>
>> I don't have opinions on the name but I do have a set of feature
>> requirements. I am currently using luaffi which needs some work to
>> remove the non portable parts, but that's not far off. Happy to switch
>> but I do need access to struct members like tables, nested structs,
>> unions, casts, metatables for structs. If there was an outline design
>> doc that would be helpful.
>
> I suggest we discuss this library in Toulouse, OK?

I think it would be very nice. But, please send me your thoughts after =).

-- 
Lourival Vieira Neto

Re: A Library for Converting Data to and from C Structs for Lua

[Lourival Vieira Neto]

On Wed, Nov 20, 2013 at 6:36 AM, Marc Balmer  wrote:
> Am 18.11.13 09:07, schrieb Marc Balmer:
>> Am 17.11.13 13:05, schrieb Marc Balmer:
>>> I came accross a small library for converting data to an from C structs
>>> for Lua, written by Roberto Ierusalimschy:
>>>
>>> http://www.inf.puc-rio.br/~roberto/struct/
>>>
>>> I plan to import it and to make it available to both lua(1) and lua(4)
>>> as follows:
>>>
>>> The source code will be imported into
>>> ${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
>>> to compile on NetBSD.
>>>
>>> Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
>>> ${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
>>> respective Makefiles etc.
>>
>> After discussion we lneto@ and others we realised that there are several
>> such libraries around, and that I as well as lneto@ wrote one.  SO we
>> decided to merge our works and have a library that allows to
>> encode/decode structured binary data in one call (my work) and that
>> allows to access individual fields as well (lneto@'s work).
>>
>> Now we need a name that covers both uses cases.  It could be "memory"
>> because it deals with memory, or just "data", which I favour.
>>
>> Opinions on the name?
>
> Since no one replied, it will go by the name 'data' and be available for
> "both" Luas.

'Data' is fine for me; I don't have a better suggestion anyway =).

> @lneto: I will start with the pack/unpack parts, you can then add your
> stuff whenever you want, ok?

Just fine for me. However, I think we need to define the API before
(re)starting coding.

Regards,
-- 
Lourival Vieira Neto

Re: vmem boundary tag manged with pool

[Lars Heidieker]

On 11/19/2013 11:02 PM, matthew green wrote:
>> I would like to change the way vmem(9) allocates boundary tags towards
>> using a pool(9) for that.
>>
>> This de-duplicates code and makes it possible to release memory pages
>> used for boundary tags.
>>
>> early patch any comments?
> 
> i fear adding another way that vmem and pool can loop, ala my
> post a couple of weeks ago:
> 
> http://mail-index.netbsd.org/tech-kern/2013/11/01/msg015881.html
> 
> 
> .mrg.
> 

thanks for taking a look.
As time allows I'm currently looking into that matter. As far as I have
analyzed it, I think that the page_allocation in uvm_km_kmem_alloc must
have failed and from there on there is a recursion within allocating a
pool_header within pool(9) see pool_grow.
The underlying problem is that we sometimes need to allocate in order to
free memory in pool(9).

The pool used for the vmem boundary tags must be using PR_PHINPAGE and
!PR_NOTOUCH so it does not allocate on freeing and allocating "only"
requires pulling a page from vmem. So no new recursion path should emerge.

But for sure you are right, this requires careful checking and the patch
is in a early stage.

Lars

in which we present an ugly hack to make sys/queue.h CIRCLEQ work

[matthew green]

hi folks.


while preparing to update to GCC 4.8 i discovered that our sys/queue.h
CIRCLEQ macros violate C aliasing rules, ultimately leading to the
compiler eliding comparisons it declared as always false.

unfortunately, changing these macros to be strictly C requires changing
the ABI of them, and while we are going to consider such a change, in
the interest of getting things working we present the following hack.
it was inspired by David Holland, and we considered placing it in
sys/cdefs.h for other consumers, but as queue.h currently only relies
on the presence of "NULL" combined with the need for ""
to work correctly for the tools build (which may be on non-netbsd
platforms.)

the visible problems are that the libc DB routines often fail, and
that nvi locks up all the time.

i'm going to commit this soon, but if anyone has more useful hacks or
real fixes, please let me/these lists know and we'll consider them.

i'm also going to purge the tree of several copies of sys/queue.h
present in 3rd party software as a follow change.

thanks to dholland, apb, joerg, martin, matt, and skrll for this
least-worst-so-far solution.


.mrg.


Index: queue.h
===
RCS file: /cvsroot/src/sys/sys/queue.h,v
retrieving revision 1.55
diff -p -r1.55 queue.h
--- queue.h 17 Jul 2013 15:50:59 -  1.55
+++ queue.h 20 Nov 2013 09:33:42 -
@@ -602,18 +602,41 @@
 /*
  * Circular queue definitions.
  */
+
+/*
+ * __launder_type():  We use this ugly hack to work around the the compiler
+ * noticing that two types may not alias each other and elide tests in code.
+ * We hit this in the CIRCLEQ macros when comparing 'struct name *' and
+ * 'struct type *' (see CIRCLEQ_HEAD()).  Modern compilers (such as GCC
+ * 4.8) declare these comparisons as always false, causing the code to
+ * not run as designed.
+ *
+ * This hack is only to be used for comparisons and thus can be fully const.
+ * Do not use for assignment.
+ *
+ * If we ever choose to change the ABI of the CIRCLEQ macros, we could fix
+ * this by changing the 
+ */
+static inline const void * __launder_type(const void *);
+static inline const void *
+__launder_type(const void *__x)
+{
+   __asm __volatile("" : "+r" (__x));
+   return __x;
+}
+
 #if defined(_KERNEL) && defined(QUEUEDEBUG)
 #define QUEUEDEBUG_CIRCLEQ_HEAD(head, field)   \
-   if ((head)->cqh_first != (void *)(head) &&  \
-   (head)->cqh_first->field.cqe_prev != (void *)(head))\
+   if ((head)->cqh_first != __launder_type(head) &&\
+   (head)->cqh_first->field.cqe_prev != __launder_type(head))  \
panic("CIRCLEQ head forw %p %s:%d", (head), \
  __FILE__, __LINE__);  \
-   if ((head)->cqh_last != (void *)(head) &&   \
-   (head)->cqh_last->field.cqe_next != (void *)(head)) \
+   if ((head)->cqh_last != __launder_type(head) && \
+   (head)->cqh_last->field.cqe_next != __launder_type(head))   \
panic("CIRCLEQ head back %p %s:%d", (head), \
  __FILE__, __LINE__);
 #define QUEUEDEBUG_CIRCLEQ_ELM(head, elm, field)   \
-   if ((elm)->field.cqe_next == (void *)(head)) {  \
+   if ((elm)->field.cqe_next == __launder_type(head)) {\
if ((head)->cqh_last != (elm))  \
panic("CIRCLEQ elm last %p %s:%d", (elm),   \
  __FILE__, __LINE__);  \
@@ -622,7 +645,7 @@
panic("CIRCLEQ elm forw %p %s:%d", (elm),   \
  __FILE__, __LINE__);  \
}   \
-   if ((elm)->field.cqe_prev == (void *)(head)) {  \
+   if ((elm)->field.cqe_prev == __launder_type(head)) {\
if ((head)->cqh_first != (elm)) \
panic("CIRCLEQ elm first %p %s:%d", (elm),  \
  __FILE__, __LINE__);  \
@@ -668,7 +691,7 @@
QUEUEDEBUG_CIRCLEQ_ELM((head), (listelm), field)\
(elm)->field.cqe_next = (listelm)->field.cqe_next;  \
(elm)->field.cqe_prev = (listelm);  \
-   if ((listelm)->field.cqe_next == (void *)(head))\
+   if ((listelm)->field.cqe_next == __launder_type(head))  \
(head)->cqh_last = (elm);   \
else\
(listelm)->field.cqe_next->field.cqe_prev = (elm);  \
@@ -680,7 +703,7 @@
QUEUEDEBUG_CIRCLEQ_ELM((h

Re: A Library for Converting Data to and from C Structs for Lua

[Marc Balmer]

Am 20.11.13 10:26, schrieb Justin Cormack:
> 
> On 20 Nov 2013 08:38, "Marc Balmer" mailto:m...@msys.ch>>
> wrote:
>> > Now we need a name that covers both uses cases.  It could be "memory"
>> > because it deals with memory, or just "data", which I favour.
>> >
>> > Opinions on the name?
>>
>> Since no one replied, it will go by the name 'data' and be available for
>> "both" Luas.
>>
>> @lneto: I will start with the pack/unpack parts, you can then add your
>> stuff whenever you want, ok?
> 
> I don't have opinions on the name but I do have a set of feature
> requirements. I am currently using luaffi which needs some work to
> remove the non portable parts, but that's not far off. Happy to switch
> but I do need access to struct members like tables, nested structs,
> unions, casts, metatables for structs. If there was an outline design
> doc that would be helpful.

I suggest we discuss this library in Toulouse, OK?

Re: A Library for Converting Data to and from C Structs for Lua

[Marc Balmer]

Am 18.11.13 09:07, schrieb Marc Balmer:
> Am 17.11.13 13:05, schrieb Marc Balmer:
>> I came accross a small library for converting data to an from C structs
>> for Lua, written by Roberto Ierusalimschy:
>>
>> http://www.inf.puc-rio.br/~roberto/struct/
>>
>> I plan to import it and to make it available to both lua(1) and lua(4)
>> as follows:
>>
>> The source code will be imported into
>> ${NETBSDSRCDIR}/sys/external/mit/struct unaltered and then be modified
>> to compile on NetBSD.
>>
>> Then ${NETBSDSRCDIR}/sys/module/luastruct/ and
>> ${NETBSDSRCDIR}/lib/lua/struct/ directories will be added with the
>> respective Makefiles etc.
> 
> After discussion we lneto@ and others we realised that there are several
> such libraries around, and that I as well as lneto@ wrote one.  SO we
> decided to merge our works and have a library that allows to
> encode/decode structured binary data in one call (my work) and that
> allows to access individual fields as well (lneto@'s work).
> 
> Now we need a name that covers both uses cases.  It could be "memory"
> because it deals with memory, or just "data", which I favour.
> 
> Opinions on the name?

Since no one replied, it will go by the name 'data' and be available for
"both" Luas.

@lneto: I will start with the pack/unpack parts, you can then add your
stuff whenever you want, ok?