Re: [Babel-users] babeld interface

[Christof Schulze]

Hi Juliusz,

On Thu, Mar 15, 2018 at 12:37:55AM +0100, Juliusz Chroboczek wrote:

> https://github.com/christf/libbabelhelper

When you feel you're ready, please provide me with a one-line 
description,

and I'll add a link to the Babel page.
I just pushed a set of improvements to the library after being inspired 
by your lexer.


The approach supports using blocking and non-blocking alike. I wanted to 
be able to parse the data in one go, which is why there are so many 
different levels of indentation in the central parsing function.


Ideas for improvements are appreciated.
Cheers

Christof


--
()  ascii ribbon campaign - against html e-mail
/\  against proprietary attachments



signature.asc
Description: PGP signature
___
Babel-users mailing list
Babel-users@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users

Re: [Babel-users] babeld interface

[Juliusz Chroboczek]

> https://github.com/christf/libbabelhelper

When you feel you're ready, please provide me with a one-line description,
and I'll add a link to the Babel page.

___
Babel-users mailing list
Babel-users@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users

Re: [Babel-users] babeld interface

[Toke Høiland-Jørgensen]

Juliusz Chroboczek  writes:

>> In this example, parsing the custom format in Python would be something like:
>
> No, parsing input using string.split is always incorrect. You really
> want to write a proper lexer.
>
> The lexer I posted above is 70 lines of C, including full error
> checking. In Python, it's probably 15 to 20 lines. Is it really worth
> switching to a bloated and unreadable format in order to avoid 20
> lines of code in each client?

Well, it is of course completely up to you where you want to set the
barrier to entry for interacting with babeld. And if you want to set
that at "being able to write a lexer", well, that is a tradeoff. Just
don't be surprised when people try to do sscanf or string.split-style
parsing anyway, and subsequently complain about the interface. But then
I guess you do like educating people, so maybe that is all well and
good... ;)

-Toke

___
Babel-users mailing list
Babel-users@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users

Re: [Babel-users] babeld interface

[Juliusz Chroboczek]

> The main benefit of a well-known format is that the requirement for a
> client goes from easy parsing code to *no* parsing code.

Right.  So either we add a dependency on JSON to babeld, or we add
a trivial parser to every program that consumes babeld's input.

> In this example, parsing the custom format in Python would be something like:

No, parsing input using string.split is always incorrect.  You really want
to write a proper lexer.

The lexer I posted above is 70 lines of C, including full error checking.
In Python, it's probably 15 to 20 lines.  Is it really worth switching to
a bloated and unreadable format in order to avoid 20 lines of code in each
client?

The problem that JSON aims to solve is not very complicated, and JSON doesn't
solve it particularly well.

-- Juliusz

___
Babel-users mailing list
Babel-users@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users

Re: [Babel-users] babeld interface

[Juliusz Chroboczek]

> I agree that the current format is very easy to parse for a human, which
> is a benefit. I am struggling to have a machine parse this though.

You need a lexer.  Here's a simple lexer for you:

  https://www.irif.fr/~jch/software/babel/babel-lexer.c

-- Juliusz

___
Babel-users mailing list
Babel-users@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users

Re: [Babel-users] babeld interface

[Toke Høiland-Jørgensen]

Juliusz Chroboczek  writes:

>> Nils and I wrote such a daemon, called mmfd. It is able to forward
>> multicast packets in the whole network. To learn about the topology and
>> the relevant neighbours, it queries babeld: mmfd is listening via
>> "monitor" on the babeld socket.
>
> Good.
>
>> What do you think of providing the same data over json to make it better
>> parsable?
>
> The format of the monitoring interface is well defined : it's a series of
> lines of the form
>
>   keyword id key value key value ...
>
> where each key/value is either space separated, or a string within double
> quotes.
>
> This format is extensible (we can add new keywords and new keys at any
> time), and it's also easy to parse by a human (Antonin and I have just
> spent much of today telnetting by hand into Babel's monitoring
> interface). So I'm not convinced that switching to JSON would have any
> benefits.

The main benefit of a well-known format is that the requirement for a
client goes from easy parsing code to *no* parsing code.

In this example, parsing the custom format in Python would be something
like:

data = {}
for line in input:
  keyword, id, rest = line.split(" ", 3)
  data[keyword] = {'id': id}
  parts = rest.split(" ") # wait, this doesn't deal with quoted values
  for key, val in zip(parts[::2], parts[1::2]):
data[keyword][key] = val

vs

data = json.loads(input)

Also, if you pretty-print the json it is quite human-readable; try e.g.
`cat datafile.json | python -m json.tool`

-Toke

___
Babel-users mailing list
Babel-users@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users

Re: [Babel-users] babeld interface

[Christof Schulze]

What do you think of providing the same data over json to make it better
parsable?


The format of the monitoring interface is well defined : it's a series of
lines of the form

 keyword id key value key value ...

where each key/value is either space separated, or a string within double
quotes.

This format is extensible (we can add new keywords and new keys at any
time), and it's also easy to parse by a human (Antonin and I have just
spent much of today telnetting by hand into Babel's monitoring interface).
So I'm not convinced that switching to JSON would have any benefits.

Please let me know if you disagree, I'm not fundamentally opposed to
adding a new command "monitor-json" if you think it's desirable, but I'd
rather avoid the added complexity.
I agree that the current format is very easy to parse for a human, which 
is a benefit. I am struggling to have a machine parse this though.


Just from the switch for babeld-1.8.0 to master a new field got 
introduced for neighbour lines. This caused my sscanf-based line-parser 
to break:


int n = sscanf(line, "%ms neighbour %*x address %ms if %ms "
"reach %x ureach %*x rxcost %d txcost %d cost %d",
, _str, , , , , 
);

The whole code for the parsing is in a library that lives here: 
https://github.com/christf/libbabelhelper

Maybe I am just not parsing this the easy but the hard way. So 
before going into the benefits of json, maybe we can see if there is a 
manufactoring error with my own approach in parsing this.


So instead of running sscanf like above, I guess I could strtok the line 
and just match for known keywords and extract their values, however 
blanks in values would be a problem to this approach. Do we really need 
to deal with blanks in values?


Christof


--
()  ascii ribbon campaign - against html e-mail
/\  against proprietary attachments



signature.asc
Description: PGP signature
___
Babel-users mailing list
Babel-users@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users

Re: [Babel-users] babeld interface

[Juliusz Chroboczek]

> Nils and I wrote such a daemon, called mmfd. It is able to forward
> multicast packets in the whole network. To learn about the topology and
> the relevant neighbours, it queries babeld: mmfd is listening via
> "monitor" on the babeld socket.

Good.

> What do you think of providing the same data over json to make it better
> parsable?

The format of the monitoring interface is well defined : it's a series of
lines of the form

  keyword id key value key value ...

where each key/value is either space separated, or a string within double
quotes.

This format is extensible (we can add new keywords and new keys at any
time), and it's also easy to parse by a human (Antonin and I have just
spent much of today telnetting by hand into Babel's monitoring interface).
So I'm not convinced that switching to JSON would have any benefits.

Please let me know if you disagree, I'm not fundamentally opposed to
adding a new command "monitor-json" if you think it's desirable, but I'd
rather avoid the added complexity.

-- Juliusz

___
Babel-users mailing list
Babel-users@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users

[Babel-users] babeld interface

[Christof Schulze]

Hello,

when building a large mesh network, Freifunk communities use maps such that 
users can find nodes in the real world.

For this there must be a daemon collecting telemetry in the network.
This daemon must know a little bit about the network topology, at least 
its neighbours.


Nils and I wrote such a daemon, called mmfd. It is able to forward 
multicast packets in the whole network. To learn about the topology and 
the relevant neighbours, it queries babeld: mmfd is listening via 
"monitor" on the babeld socket.


This works reasonably well as long as the string output format of the 
babel socket does not change.


What do you think of providing the same data over json to make it better 
parsable?


Regards

Christof


--
()  ascii ribbon campaign - against html e-mail
/\  against proprietary attachments



signature.asc
Description: PGP signature
___
Babel-users mailing list
Babel-users@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/babel-users