Re: libnv
> On Aug 27, 2018, at 2:25 PM, David Holland wrote: > > On Mon, Aug 27, 2018 at 08:41:43AM -0700, Jason Thorpe wrote: >>> [proplib] >> >> Let me try to address these points one by one: > > Now that you're back, can you explain a few more basic points about > proplib? I'll do my best to remember :-) > (a) What's its mission? Is it supposed to be a data *storage* library, > that is, the data lives in proplib and you do proplib accesses to > extract the bits of it you want as you need them? Or is it supposed to > be a data *transfer* library, that is, the data lives somewhere else > and the expectation is that when you receive a proplib bundle that you > immediately unpack it into its own data structures? (Note that saving > to disk and loading back later is roughly comparable to sending and > receiving among entities -- that's not what this question is about.) It can be either. Consider the case of properties on device_t's ... it's the storage medium for that information. Or consider the case of a configuration file ... this isn't really the same thing as "saving state out to disk and reading it back in later", it's one application serializing the information in a standard way to be consumed by another ... and how the consumer deals with it is really up to them. Or consider the case of wanting to collect some information from the kernel for consumption by userland (for display purposes, or whatever) ... but you want to isolate the receiver from the implementation details of the kernel sender. Some additional utility functions could make it easier to use in various situations, but it's intent is to be "useful wherever" > (b) What's its data model? Is it supposed to handle hierarchical > tree-structured data of the kind you might find in an ordinary > programming language's native data types? Or is it supposed to support > graph-structured data where many locations can share references to > common values or sub-objects? The former is how it's generally used, but it shares references to common sub-objects as a memory saving measure (either by assuming you use the API that manipulates the objects directly and do this yourself, or by de-duping certain immutable objects, as with prop_number_t). There is not attempt to detect cycles, however, so y'know, exercise caution. > (c) And also, what's its data model? What was the criterion used to > determine which atom types it supports? The data types supported by Apple XML property lists, with the following caveats: -- No attempt to support or . There is no reason those could not be added, but their use in standalone and kernel environments is not feasible; they could certainly be excluded based on #ifdef... -- I serialized unsigned numbers as hex (and assume that hex numbers that are read in are unsigned), and serialize signed as decimal. I believe macOS will read both decimal and hex, but always writes out in decimal. I think. I should double-check. I captured bool as a separate object type, as in CoreFoundation (CFBoolean), rather than in the number object as in Foundation (NSNumber). -- thorpej
Re: libnv
On Mon, Aug 27, 2018 at 08:41:43AM -0700, Jason Thorpe wrote: > > [proplib] > > Let me try to address these points one by one: Now that you're back, can you explain a few more basic points about proplib? (a) What's its mission? Is it supposed to be a data *storage* library, that is, the data lives in proplib and you do proplib accesses to extract the bits of it you want as you need them? Or is it supposed to be a data *transfer* library, that is, the data lives somewhere else and the expectation is that when you receive a proplib bundle that you immediately unpack it into its own data structures? (Note that saving to disk and loading back later is roughly comparable to sending and receiving among entities -- that's not what this question is about.) (b) What's its data model? Is it supposed to handle hierarchical tree-structured data of the kind you might find in an ordinary programming language's native data types? Or is it supposed to support graph-structured data where many locations can share references to common values or sub-objects? (c) And also, what's its data model? What was the criterion used to determine which atom types it supports? -- David A. Holland dholl...@netbsd.org
Re: libnv
> On Aug 27, 2018, at 7:49 AM, Mindaugas Rasiukevicius wrote: > I do not think there was any conclusion either, or a conclusion that > we always want to have static serialisation with a schema (IMO, there > are pros and cons in both cases). Worth noting that there are multiple > libraries already: we already imported libnvpair (with tons of other > code for ZFS and DTrace) and nobody complained; we still have XDR. XDR has a separate reason for existing -- it is enshrined in wire protocols. > While libnv is not a panacea, it solves some actual problems: it has > intuitive reference counting (symmetric API), it supports accumulated > error handling, it provides more efficient binary encoding, it does not > have shared structures with internal locking, it does not have awkward > API. As a side note, it also makes code sharing with FreeBSD (as well > as illumos) much easier. Meanwhile, there has been little action on > improving proplib or proposing a tangible replacement of it. Let me try to address these points one by one: ==> It has intuitive reference counting (symmetric API) I just sent you a separate mail on this subject. Actually, the libprop reference counting rules are very simple: "If you create an object, you implicitly own the reference and you are responsible for disposing of it. In all other cases, references must be explicitly retained." That's it. It's really not that complicated. Containers (arrays and dictionaries) retain their own references explicitly (because they want the object the live on not matter what the caller does with its own reference). When you "get" an object from the container, you did not create it, and thus need to explicitly retain your own (that you, of course, are responsible for disposing of later). Please explain to me how this referencing counting rule is non-intuitive. ==> It supports accumulated error handling I personally find APIs like this kind of annoying, but if you find it valuable, it could be added to libprop with a utility wrapper API. ==> It provides more efficient binary encoding I'll grant this one, but with a caveat -- I always intended to to add the "binary plist" serialization format to libprop, but never got around to it. It would require a refactor of the serialization stuff that's already in libprop, but this is a solvable problem and have been thinking of doing it soonish anyway because of some other ideas I've been toying with in my head. ==> It does not have shared structures with internal locking I'll grant that this was probably a mistake initially with dictionaries and arrays. Serialization for these should be handled at the caller's level, because in many instances you would need that anyway if you expect multiple threads to be accessing the container (to provide correct semantics at the caller's layer). So, as such, it would probably be worth having a discussion about whether removing the internal locking would be a safe thing to do. ==> It does not have an awkward API The individual object APIs themselves can be kind of awkward to use, but that was the price to pay for flexibility. There are already several utility APIs that provide a simpler interface, and those could be extended. > > Anyway, I am not proposing that everybody should use libnv. However, > it is is quite lightweight and self-contained. I am already using libnv > in the upstream NPF. The conversion to libnv made the code simpler, > shorter and easier to read. And I am not planning to maintain proplib > version: it causes too much headache and the differences are too vast. > > -- > Mindaugas -- thorpej
Re: libnv
Kamil Rytarowski wrote: > On 27.08.2018 02:39, Mindaugas Rasiukevicius wrote: > > Hi, > > > > Here is patch to import the FreeBSD's libnv library: > > > > http://www.netbsd.org/~rmind/libnv.diff > > > > For more details on libnv: > > > > https://www.freebsd.org/cgi/man.cgi?query=nv=9 > > > > I have already converted the standalone/upstream NPF to use libnv, > > so it would be the first user of this library. > > > > Thanks. > > > > What's the rationale for this library? Is it an alternative for proplib? > Yes, see my reply to Joerg's email in the thread. -- Mindaugas
Re: libnv
Joerg Sonnenberger wrote: > > > > Here is patch to import the FreeBSD's libnv library: > > > > http://www.netbsd.org/~rmind/libnv.diff > > I don't think anything has changed since the last time this was > discussed, so I consider this a really, really wrong idea going forward. > The only advantage libnv has over proplib is being somewhat simpler. It > doesn't address any other concern like lack of schemas. I do not think there was any conclusion either, or a conclusion that we always want to have static serialisation with a schema (IMO, there are pros and cons in both cases). Worth noting that there are multiple libraries already: we already imported libnvpair (with tons of other code for ZFS and DTrace) and nobody complained; we still have XDR. While libnv is not a panacea, it solves some actual problems: it has intuitive reference counting (symmetric API), it supports accumulated error handling, it provides more efficient binary encoding, it does not have shared structures with internal locking, it does not have awkward API. As a side note, it also makes code sharing with FreeBSD (as well as illumos) much easier. Meanwhile, there has been little action on improving proplib or proposing a tangible replacement of it. Anyway, I am not proposing that everybody should use libnv. However, it is is quite lightweight and self-contained. I am already using libnv in the upstream NPF. The conversion to libnv made the code simpler, shorter and easier to read. And I am not planning to maintain proplib version: it causes too much headache and the differences are too vast. -- Mindaugas
Re: libnv
On Mon, Aug 27, 2018 at 01:39:38AM +0100, Mindaugas Rasiukevicius wrote: > Hi, > > Here is patch to import the FreeBSD's libnv library: > > http://www.netbsd.org/~rmind/libnv.diff I don't think anything has changed since the last time this was discussed, so I consider this a really, really wrong idea going forward. The only advantage libnv has over proplib is being somewhat simpler. It doesn't address any other concern like lack of schemas. Joerg
Re: libnv
On 27.08.2018 02:39, Mindaugas Rasiukevicius wrote: > Hi, > > Here is patch to import the FreeBSD's libnv library: > > http://www.netbsd.org/~rmind/libnv.diff > > For more details on libnv: > > https://www.freebsd.org/cgi/man.cgi?query=nv=9 > > I have already converted the standalone/upstream NPF to use libnv, > so it would be the first user of this library. > > Thanks. > What's the rationale for this library? Is it an alternative for proplib? signature.asc Description: OpenPGP digital signature
