Re: [libvirt] Should we switch to a different JSON library?

2018-02-14 Thread Pino Toscano 
On Monday, 12 February 2018 19:47:00 CET Ján Tomko wrote:
> On Mon, Feb 12, 2018 at 02:38:02PM +0100, Pino Toscano wrote:
> >On Tuesday, 7 November 2017 14:05:25 CET Martin Kletzander wrote:
> >>  - Jansson [3] - I really like this one.  The API seems very intuitive,
> >>  it has nice documentation [4] in readthedocs (and I'm
> >>  not talking about the visual style, but how easy is to
> >>  find information), it can be used for formatting JSON
> >>  in a similar way we are doing it.  It has json_auto_t
> >>  (optional) type that uses the attribute cleanup for
> >>  automatic scope dereference (just in case we want to
> >>  use it), it has iterators... did I tell you I like this
> >>  one a lot?
> >>
> >> What do you (others) think of switching the JSON library?  Do you know
> >> about any other projects that could be used considering license,
> >> platform support, etc.?  Also feel free to fix any mistakes I might have
> >> posted.  I double-checked it, but you know, "trust, but verify".
> >
> >FYI: libguestfs just switched to Jansson [1], so any version starting
> >from 1.39.1 will use it instead of Yajl.  In case of libguestfs, yajl
> >was used directly, without wrappers of any sort, and the switch was
> >straightforward.
> >
> >[1] 
> >https://github.com/libguestfs/libguestfs/commit/bd1c5c9f4dcf38458099db8a0bf4659a07ef055d
> >
> 
> I do have a working virjson.c implementation in my local git waiting to be
> polished and sent. The issues with libvirt were:
> * virjson.c storing numbers as a string
> * backwards compatibility (AFAIK we want to support building on
>   RHEL/CentOS 6, which did not have recent enough jansson - for the
>   _foreach macros, at least 2.5 is needed)
>   If we really need to maintain two implementations side-by-side,
>   one of them should have an expiration date.

RHEL/CentOS 6 do not seem to have jansson -- it's in EPEL, and its
version there is 2.9.
RHEL/CentOS 7 have jansson 2.10.

> I don't see any version check in that libguestfs commit, what are the
> compatibility requirements there?

I though the APIs we used were old enough, but apparently not for an
old distro I still had around (Mageia 5, EOL).  I just bumped the
minimum requirement in libguestfs to 2.7, which is old enough, and
IMHO good enough as baseline; see
https://repology.org/metapackage/jansson/versions

-- 
Pino Toscano

signature.asc
Description: This is a digitally signed message part.
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] Should we switch to a different JSON library?

2018-02-12 Thread Daniel P . Berrangé 
On Mon, Feb 12, 2018 at 07:47:00PM +0100, Ján Tomko wrote:
> On Mon, Feb 12, 2018 at 02:38:02PM +0100, Pino Toscano wrote:
> > On Tuesday, 7 November 2017 14:05:25 CET Martin Kletzander wrote:
> > >  - Jansson [3] - I really like this one.  The API seems very intuitive,
> > >  it has nice documentation [4] in readthedocs (and I'm
> > >  not talking about the visual style, but how easy is to
> > >  find information), it can be used for formatting JSON
> > >  in a similar way we are doing it.  It has json_auto_t
> > >  (optional) type that uses the attribute cleanup for
> > >  automatic scope dereference (just in case we want to
> > >  use it), it has iterators... did I tell you I like this
> > >  one a lot?
> > > 
> > > What do you (others) think of switching the JSON library?  Do you know
> > > about any other projects that could be used considering license,
> > > platform support, etc.?  Also feel free to fix any mistakes I might have
> > > posted.  I double-checked it, but you know, "trust, but verify".
> > 
> > FYI: libguestfs just switched to Jansson [1], so any version starting
> > from 1.39.1 will use it instead of Yajl.  In case of libguestfs, yajl
> > was used directly, without wrappers of any sort, and the switch was
> > straightforward.
> > 
> > [1] 
> > https://github.com/libguestfs/libguestfs/commit/bd1c5c9f4dcf38458099db8a0bf4659a07ef055d
> > 
> 
> I do have a working virjson.c implementation in my local git waiting to be
> polished and sent. The issues with libvirt were:
> * virjson.c storing numbers as a string
> * backwards compatibility (AFAIK we want to support building on
>  RHEL/CentOS 6, which did not have recent enough jansson - for the
>  _foreach macros, at least 2.5 is needed)
>  If we really need to maintain two implementations side-by-side,
>  one of them should have an expiration date.

IMHO we only need to keepp yajl if RHEL6 doesn't have suitable jansson.
So ideally we'd figure out how to get that working, so we can ditch
yajl straightaway.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] Should we switch to a different JSON library?

2018-02-12 Thread Ján Tomko 

On Mon, Feb 12, 2018 at 02:38:02PM +0100, Pino Toscano wrote:

On Tuesday, 7 November 2017 14:05:25 CET Martin Kletzander wrote:

 - Jansson [3] - I really like this one.  The API seems very intuitive,
 it has nice documentation [4] in readthedocs (and I'm
 not talking about the visual style, but how easy is to
 find information), it can be used for formatting JSON
 in a similar way we are doing it.  It has json_auto_t
 (optional) type that uses the attribute cleanup for
 automatic scope dereference (just in case we want to
 use it), it has iterators... did I tell you I like this
 one a lot?

What do you (others) think of switching the JSON library?  Do you know
about any other projects that could be used considering license,
platform support, etc.?  Also feel free to fix any mistakes I might have
posted.  I double-checked it, but you know, "trust, but verify".


FYI: libguestfs just switched to Jansson [1], so any version starting
from 1.39.1 will use it instead of Yajl.  In case of libguestfs, yajl
was used directly, without wrappers of any sort, and the switch was
straightforward.

[1] 
https://github.com/libguestfs/libguestfs/commit/bd1c5c9f4dcf38458099db8a0bf4659a07ef055d



I do have a working virjson.c implementation in my local git waiting to be
polished and sent. The issues with libvirt were:
* virjson.c storing numbers as a string
* backwards compatibility (AFAIK we want to support building on
 RHEL/CentOS 6, which did not have recent enough jansson - for the
 _foreach macros, at least 2.5 is needed)
 If we really need to maintain two implementations side-by-side,
 one of them should have an expiration date.

I don't see any version check in that libguestfs commit, what are the
compatibility requirements there?

Jan


signature.asc
Description: Digital signature
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] Should we switch to a different JSON library?

2018-02-12 Thread Pino Toscano 
On Tuesday, 7 November 2017 14:05:25 CET Martin Kletzander wrote:
>  - Jansson [3] - I really like this one.  The API seems very intuitive,
>  it has nice documentation [4] in readthedocs (and I'm
>  not talking about the visual style, but how easy is to
>  find information), it can be used for formatting JSON
>  in a similar way we are doing it.  It has json_auto_t
>  (optional) type that uses the attribute cleanup for
>  automatic scope dereference (just in case we want to
>  use it), it has iterators... did I tell you I like this
>  one a lot?
> 
> What do you (others) think of switching the JSON library?  Do you know
> about any other projects that could be used considering license,
> platform support, etc.?  Also feel free to fix any mistakes I might have
> posted.  I double-checked it, but you know, "trust, but verify".

FYI: libguestfs just switched to Jansson [1], so any version starting
from 1.39.1 will use it instead of Yajl.  In case of libguestfs, yajl
was used directly, without wrappers of any sort, and the switch was
straightforward.

[1] 
https://github.com/libguestfs/libguestfs/commit/bd1c5c9f4dcf38458099db8a0bf4659a07ef055d

-- 
Pino Toscano

signature.asc
Description: This is a digitally signed message part.
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] Should we switch to a different JSON library?

2017-11-07 Thread Neal Gompa 
On Tue, Nov 7, 2017 at 9:34 AM, Martin Kletzander  wrote:
> On Tue, Nov 07, 2017 at 01:38:51PM +, Richard W.M. Jones wrote:
>>
>> On Tue, Nov 07, 2017 at 01:28:00PM +, Daniel P. Berrange wrote:
>>>
>>> On Tue, Nov 07, 2017 at 02:05:25PM +0100, Martin Kletzander wrote:
>>> >  4) The are workarounds for it in the code
>>>
>>> I don't recall what these are now ?
>>
>>
>
> I shouldn't have said 'workarounds', I meant all the conditioning for
> differentiation between yajl 1 and 2 and supporting both version with
> pkg-config
> and without.
>
>> libguestfs needs to disable some warnings around yajl code:
>>
>>  /* GCC can't work out that the YAJL_IS_ test is sufficient to
>>   * ensure that YAJL_GET_ later doesn't return NULL.
>>   */
>>  #if defined(__GNUC__) && __GNUC__ >= 6 /* gcc >= 6 */
>>  #pragma GCC diagnostic ignored "-Wnull-dereference"
>>  #endif
>>
>> However we don't need any other workarounds.
>>
>> As you say yajl "just works".
>>
>> It might however be a problem in future if JSON itself was changed,
>> eg. new types, fix the trailing comma problem, etc.  We'd want a more
>> responsive upstream to handle that.
>>
>> - - -
>>
>> Martin, did you look at the dependencies of these other libraries?
>> If they depend on other stuff, that could make them harder to use.
>> Also are they all available in at least Fedora, Debian, OpenSUSE and
>> Arch/AUR?
>>
>
> Just did, all of these were successful:
>
> docker run -it fedora dnf search jansson
> docker run -it opensuse zypper search jansson
> docker run -it base/archlinux pacman -Sys jansson
> docker run -it ubuntu sh -c "apt update; apt search jansson"
> docker run -it debian sh -c "apt update; apt search jansson"
>
> Even:
>
> docker run -it debian:stable sh -c "apt update; apt search jansson"
>
> found version 2.9 which is the second latest one, I believe.
>
>

I used Repology to check jansson:
https://repology.org/metapackage/jansson/versions

It looks like the range of supported versions is v2.7 to v2.10.

I see no reason it can't be used.

-- 
真実はいつも一つ！/ Always, there's only one truth!

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] Should we switch to a different JSON library?

2017-11-07 Thread Martin Kletzander 

On Tue, Nov 07, 2017 at 01:28:00PM +, Daniel P. Berrange wrote:

On Tue, Nov 07, 2017 at 02:05:25PM +0100, Martin Kletzander wrote:

Hi everyone,

so we are using yajl for parsing JSON.  However there are some reasons
why we might consider switching to another one:

 1) It is basically dead upstream


One could say that it is simply "feature complete" upstream so doesn't
need more work, but none the less, I have a general bad feeling about
relying on a library with no active maint work going on for best part
of 4 years, and 68 open pull requests.


 2) We're just using the lexer part of it
 3) We only use it for parsing


Aren't 2 & 3 the same point repeated ? Regardless though, we use yajl
for formatting JSON too (virJSONValueToString)



I meant 2 different things.  By only using the parser I meant that for
some reason (I guess the first implementation wasn't good enough, but
you'll be the one to know the proper one) we store it in our own object
for which we need to parse the data similarly to how SAX works, right?
We register callbacks and we do lot of stuff for each value found.  It
looks like jansson has enough APIs so that virJSON* could be just
trivial wrappers around json_* functions/structures and for example for
parsing we could just call json_load().  I haven't studied it
thoroughly, but that's the feeling I got.

The second thing was a bit of a misunderstanding on my part, we clearly
use it for formatting as well, yes.


 4) The are workarounds for it in the code


I don't recall what these are now ?


So I looked at some options and found few other libraries, I only took
those that are widely available (read: I checked if some random
downstream distro has them), however most of them were not very much
usable.  Except one.  But here's the list of few others that didn't look
that bad either.  All are MIT-licensed, try to be thread-safe and
support pkg-config:

- libfastjson [1] - from rsyslog community, optimized for working with
logs, used almost only by rsyslog, it's supposed to
have doxygen docs, they switched to libfastjson
from json-c due to some problems with it
(performance, ref counting bug, ...)


If its only used by 1 app, I would stay well clear of it, as that's just
one step away from be unused and dead.


- json-c [2] - looked abandoned until I looked at the proper github
   page, "documentation" in doxygen


Yeah, this is the main one I know about and existed way back when I
picked yajl originally.

It has plenty of apps using it in Fedora, which is good.

At first the header files look to have good namespacing, but then I
see horrific things like this polluting the namespaces:

 #undef FALSE
 #define FALSE ((json_bool)0)

 #undef TRUE
 #define TRUE ((json_bool)1)

 #define hexdigit(x) (((x) <= '9') ? (x) - '0' : ((x) & 7) + 9)
 #define error_ptr(error) ((void*)error)
 #define error_description(error)  (json_tokener_errors[error])
 #define is_error(ptr) (ptr == NULL)


- Jansson [3] - I really like this one.  The API seems very intuitive,
it has nice documentation [4] in readthedocs (and I'm
not talking about the visual style, but how easy is to
find information), it can be used for formatting JSON
in a similar way we are doing it.  It has json_auto_t
(optional) type that uses the attribute cleanup for
automatic scope dereference (just in case we want to
use it), it has iterators... did I tell you I like this
one a lot?


Like yajl did, they seem to have had an explicit ABI break at their 2.0
version, which was released in 2011. Not a big deal if we only support
their 2.x series, especially if we keep yajl as an optional fallback

It has 15 open PRs and 32 open bugs, but unlike yajl it actually has
active development taking place

Its header namespace looks clean too.


What do you (others) think of switching the JSON library?  Do you know
about any other projects that could be used considering license,
platform support, etc.?  Also feel free to fix any mistakes I might have
posted.  I double-checked it, but you know, "trust, but verify".


I'd be broadly in favour of switching to something that is not dead :-)

I agree that Jansson appears the most promising.

I think we would have to keep yajl support in parallel, but that's not
too hard, since this is all isolated in virjson.c



Sure, I guess we would just have 2 implementations like we used to have
with threads, selecting particular one at compile-time.


Regards,
Daniel
--
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|


signature.asc
Description: Digital signature
--
libvir-list mailing list
libvir-list@redhat.com

Re: [libvirt] Should we switch to a different JSON library?

2017-11-07 Thread Martin Kletzander 

On Tue, Nov 07, 2017 at 01:38:51PM +, Richard W.M. Jones wrote:

On Tue, Nov 07, 2017 at 01:28:00PM +, Daniel P. Berrange wrote:

On Tue, Nov 07, 2017 at 02:05:25PM +0100, Martin Kletzander wrote:
>  4) The are workarounds for it in the code

I don't recall what these are now ?




I shouldn't have said 'workarounds', I meant all the conditioning for
differentiation between yajl 1 and 2 and supporting both version with pkg-config
and without.


libguestfs needs to disable some warnings around yajl code:

 /* GCC can't work out that the YAJL_IS_ test is sufficient to
  * ensure that YAJL_GET_ later doesn't return NULL.
  */
 #if defined(__GNUC__) && __GNUC__ >= 6 /* gcc >= 6 */
 #pragma GCC diagnostic ignored "-Wnull-dereference"
 #endif

However we don't need any other workarounds.

As you say yajl "just works".

It might however be a problem in future if JSON itself was changed,
eg. new types, fix the trailing comma problem, etc.  We'd want a more
responsive upstream to handle that.

- - -

Martin, did you look at the dependencies of these other libraries?
If they depend on other stuff, that could make them harder to use.
Also are they all available in at least Fedora, Debian, OpenSUSE and
Arch/AUR?



Just did, all of these were successful:

docker run -it fedora dnf search jansson
docker run -it opensuse zypper search jansson
docker run -it base/archlinux pacman -Sys jansson
docker run -it ubuntu sh -c "apt update; apt search jansson"
docker run -it debian sh -c "apt update; apt search jansson"

Even:

docker run -it debian:stable sh -c "apt update; apt search jansson"

found version 2.9 which is the second latest one, I believe.


Rich.

--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/


signature.asc
Description: Digital signature
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] Should we switch to a different JSON library?

2017-11-07 Thread Richard W.M. Jones 
On Tue, Nov 07, 2017 at 01:28:00PM +, Daniel P. Berrange wrote:
> On Tue, Nov 07, 2017 at 02:05:25PM +0100, Martin Kletzander wrote:
> >  4) The are workarounds for it in the code
> 
> I don't recall what these are now ?

libguestfs needs to disable some warnings around yajl code:

  /* GCC can't work out that the YAJL_IS_ test is sufficient to
   * ensure that YAJL_GET_ later doesn't return NULL.
   */
  #if defined(__GNUC__) && __GNUC__ >= 6 /* gcc >= 6 */
  #pragma GCC diagnostic ignored "-Wnull-dereference"
  #endif

However we don't need any other workarounds.

As you say yajl "just works".

It might however be a problem in future if JSON itself was changed,
eg. new types, fix the trailing comma problem, etc.  We'd want a more
responsive upstream to handle that.

- - -

Martin, did you look at the dependencies of these other libraries?
If they depend on other stuff, that could make them harder to use.
Also are they all available in at least Fedora, Debian, OpenSUSE and
Arch/AUR?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] Should we switch to a different JSON library?

2017-11-07 Thread Daniel P. Berrange 
On Tue, Nov 07, 2017 at 02:05:25PM +0100, Martin Kletzander wrote:
> Hi everyone,
> 
> so we are using yajl for parsing JSON.  However there are some reasons
> why we might consider switching to another one:
> 
>  1) It is basically dead upstream

One could say that it is simply "feature complete" upstream so doesn't
need more work, but none the less, I have a general bad feeling about
relying on a library with no active maint work going on for best part
of 4 years, and 68 open pull requests.

>  2) We're just using the lexer part of it
>  3) We only use it for parsing

Aren't 2 & 3 the same point repeated ? Regardless though, we use yajl
for formatting JSON too (virJSONValueToString)

>  4) The are workarounds for it in the code

I don't recall what these are now ?

> So I looked at some options and found few other libraries, I only took
> those that are widely available (read: I checked if some random
> downstream distro has them), however most of them were not very much
> usable.  Except one.  But here's the list of few others that didn't look
> that bad either.  All are MIT-licensed, try to be thread-safe and
> support pkg-config:
> 
> - libfastjson [1] - from rsyslog community, optimized for working with
> logs, used almost only by rsyslog, it's supposed to
> have doxygen docs, they switched to libfastjson
> from json-c due to some problems with it
> (performance, ref counting bug, ...)

If its only used by 1 app, I would stay well clear of it, as that's just
one step away from be unused and dead.

> - json-c [2] - looked abandoned until I looked at the proper github
>page, "documentation" in doxygen

Yeah, this is the main one I know about and existed way back when I
picked yajl originally.

It has plenty of apps using it in Fedora, which is good.

At first the header files look to have good namespacing, but then I
see horrific things like this polluting the namespaces:

  #undef FALSE
  #define FALSE ((json_bool)0)

  #undef TRUE
  #define TRUE ((json_bool)1)

  #define hexdigit(x) (((x) <= '9') ? (x) - '0' : ((x) & 7) + 9)
  #define error_ptr(error) ((void*)error)
  #define error_description(error)  (json_tokener_errors[error])
  #define is_error(ptr) (ptr == NULL)

> - Jansson [3] - I really like this one.  The API seems very intuitive,
> it has nice documentation [4] in readthedocs (and I'm
> not talking about the visual style, but how easy is to
> find information), it can be used for formatting JSON
> in a similar way we are doing it.  It has json_auto_t
> (optional) type that uses the attribute cleanup for
> automatic scope dereference (just in case we want to
> use it), it has iterators... did I tell you I like this
> one a lot?

Like yajl did, they seem to have had an explicit ABI break at their 2.0
version, which was released in 2011. Not a big deal if we only support
their 2.x series, especially if we keep yajl as an optional fallback

It has 15 open PRs and 32 open bugs, but unlike yajl it actually has
active development taking place

Its header namespace looks clean too.

> What do you (others) think of switching the JSON library?  Do you know
> about any other projects that could be used considering license,
> platform support, etc.?  Also feel free to fix any mistakes I might have
> posted.  I double-checked it, but you know, "trust, but verify".

I'd be broadly in favour of switching to something that is not dead :-)

I agree that Jansson appears the most promising.

I think we would have to keep yajl support in parallel, but that's not
too hard, since this is all isolated in virjson.c

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] Should we switch to a different JSON library?

2017-11-07 Thread Martin Kletzander 

Hi everyone,

so we are using yajl for parsing JSON.  However there are some reasons
why we might consider switching to another one:

 1) It is basically dead upstream
 2) We're just using the lexer part of it
 3) We only use it for parsing
 4) The are workarounds for it in the code

So I looked at some options and found few other libraries, I only took
those that are widely available (read: I checked if some random
downstream distro has them), however most of them were not very much
usable.  Except one.  But here's the list of few others that didn't look
that bad either.  All are MIT-licensed, try to be thread-safe and
support pkg-config:

- libfastjson [1] - from rsyslog community, optimized for working with
logs, used almost only by rsyslog, it's supposed to
have doxygen docs, they switched to libfastjson
from json-c due to some problems with it
(performance, ref counting bug, ...)

- json-c [2] - looked abandoned until I looked at the proper github
   page, "documentation" in doxygen

- Jansson [3] - I really like this one.  The API seems very intuitive,
it has nice documentation [4] in readthedocs (and I'm
not talking about the visual style, but how easy is to
find information), it can be used for formatting JSON
in a similar way we are doing it.  It has json_auto_t
(optional) type that uses the attribute cleanup for
automatic scope dereference (just in case we want to
use it), it has iterators... did I tell you I like this
one a lot?

What do you (others) think of switching the JSON library?  Do you know
about any other projects that could be used considering license,
platform support, etc.?  Also feel free to fix any mistakes I might have
posted.  I double-checked it, but you know, "trust, but verify".

Have a nice day,
Martin

[1] https://github.com/rsyslog/libfastjson
[2] https://github.com/json-c/json-c
[3] http://www.digip.org/jansson/
[4] https://jansson.readthedocs.io/


signature.asc
Description: Digital signature
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list