Michael, Thank you for all of your work on this. It is good to see someone approaching this from an engineering perspective rather than a "let's throw something together" point of view. (Unlike the interpreter changes to add subroutines and flow of control code that I wrote.)
On 2/7/2013 7:46 AM, Michael Haberler wrote: > I assume you have suffered through the precursor to this writeup, 'a > mini-tutorial on the flow of status information in LinuxCNC'. > > -- > > Before I start, let me point out the following facts in the example I used to > describe the flow of status information: > > - we started with a HAL pin > - we ended up with a Python attribute. > > Now both are 'late binding' - most of you are used to defining HAL pins in > components, and how pins are glued together at _startup time_. Also many of > you are familiar with Python programming; that isnt compiled either - reedit, > reexecute, done. > > Now remember the steps from motion 'probe-input' to Python 'probe_input'. We > went through 2 data structures - emcmot_status_t and EMC_MOT_STAT to convey > that single bit. Unfortunately, these data structures are compiled with the C > and C++ compilers respectively - that's called 'early binding' and it > translates into the following limitation: > > The type and layout of information bits which can be conveyed through an > NML message is fixed at compile time. > > So no matter how flexible HAL or Python is - there you go: either them > LinuxCNC gurus have forethought your use case for an extra pin value in a NML > message, or you're out on the limb. Can't be done without without several > nontrivial changes here and there, and it's pretty hard to get right first > time, nevermind having it supported in mainline code. Thats why nobody does > it, and it is also the reason why emcmot_status_t and EMC_MOT_STAT have > become the kitchen sink of status information: just in case somebody might > need it, keep it there. Result: several incomprehensible whales of > datastructures. > > Now assume a non-LinuxCNC application of HAL/RTAPI for a moment - just for > the sake of argument, assume the prototypical 'garden sprinkler' application. > Its is safe to assume that the user will find HAL to be super-flexible, but > be less than enthusiastic about the set of NML messages provided to talk to > it, and report status from it - assuming the sprinkler controller is some > headless board, maybe a Beaglebone. Trajectory status doesnt make a whole lot > of sense when reporting whether a faucet is open or closed. > > Standing back a bit, it's like we've just shot ourselves in the foot big > time: we have super-flexible languages for GUI's, we have HAL and its > stunning arsenal of components, and we have a veritable straightjacket of a > limitation when communicating between the two. No wonder folks hit upon the > idea they could circumvent the NML restrictions by including HAL access in > UI's, and thats where we are with PyVCP, and GladeVCP: we have a very > restricted mechanism to communicate which is in principle network capable, > and a much more generic mechanism which is not network capable, and we have > _both of them in parallel_, inheriting the union of restrictions derived from > both. > > I'm sorry, this cannot be it. So we arrive at requirement #1: > > Any future NML replacement should be late-binding capable. YES. Although we should not ignore the fact that this will be inherently slower and take more memory than the compiled (early binding) approach. If this were an engineering requirements document, I would use LARGE TYPE to remind the reader that the existing architecture is (I'm guessing here) over thirty years old and that since that time, processors have increased in memory by a factor of 1000 and in speed by a factor of 1000. During that time, our control loop requirements have increased in speed by perhaps a factor of ten. (Feel free to provide more precise numbers -- I consider an order of magnitude to be a minor error here.) > > That means that message contents become subject to integrator definition and > extension, as opposed to guru discretion. Note that I'm not suggesting > everybody _has_ to do this; to the contrary, common message types will surely > be held in some library, and reused. It is all about enabling it, not forcing > it. > > ---- > > Let's look for a moment what type of information is reported from HAL > components to task and other using code: it turns out > > It is all scalar values: floats, signed ints, unsigned ints, booleans. > > (there are some minor exceptions like the gcode filename which are not > relevant here). Might not be relevant HERE, but should (IMHO) be supported. > > That maps perfectly on the HAL primitive datatypes. However, an essential > feature of NML is to arrange single scalar values into a compound structure, > like EMC_MOTION_STAT or EMC_IO_STAT, and to report those structures 'in one > go'. HAL as it stands doesnt support that. > > So we arrive at requirement #2: > > Any future NML replacement must support grouping of scalars into compound > messages. > > That brings the question: how and where is that grouping done? In essence you > need a way of expressing 'I want message X to contain values A, B and C, > which are of type int, float and bool'. That's a typical language feature: C > does it, C++ does it, Python does it, Google Protobufs does it, unfortunately > HAL/halcmd does not. > > So the options are: a) use an existing language to do the grouping b) extend > HAL to support grouping. > > a) has been the NML approach, with the consequences outlined in the previous > article. Whatever the language: it is a second language, which must be > learned and understood, and brings its own set of restrictions. > b) is new and requires some thought. > > If route b) is chosen, it would be wise to get away with the very minimum > bloat of HAL per se, and do the rest elsewhere - in using code, like > components. So lets think though this a bit. > > Object grouping in HAL > ---------------------- > > In HAL right now we have the following objects: pins, signals, params, > threads, modules, thread functions. > > Assume we add to the above two new objects: a 'group', and a 'member' object > - both of them having a HAL name, a type, and several attributes, just like > the rest of them. > > HAL groups would have the following attributes: > - a name and a uniqe integer ID > - some, say 2, integer attributes, which are not interpreted by HAL per se > but are for 'using code only'. > - an ordered list of members. > > A HAL member would have the following attributes: > - any member is a part of a single given group. I assume you mean, "at most one group at a time." > - a member refers to a HAL signal, which must exist at the time when the > member is added to a group. > - again, say 2, attributes; an integer and a float value; again not > interpreted by HAL per se but by using code only. > > To support groups and members, the HAL C library needs a bit of extension - > adding, deleting or iterating over groups; adding and deleting members. > Also halcmd needs to understand the new objects, and be able to display them. > > Smells like a spec, so let's do > > A simple usage example: defining a group in halcmd: > --------------------------------------------------- > > We use 'monitoring a battery-backed power supply' as example scenario. > > Assume a meaningful status report for such a power supply would contain: > current voltage, current load, is the mains fuse blown/OK, are we're running > on battery, or on mains. > > Now let's translate that into a HAL group: > > ----- extended halcmd ---- > # declare the signals - these must exist before using them in a member > newsig volt float > newsig amps float > newsig mains bit > newsig fuse-ok bit > > # here we would link any pins reporting actual values from the power supply > # to the signals declared at the top > net volt meter.voltage > # etc etc > > # declare named group 'power-suply' > # groups have a unique ID (23 below), plus two optional userargs (10 and > 1000). > # HAL doesnt care about userargs, except they be integer > newg power-supply 23 10 1000 > > # add some members to this group: > # note members can have an optional int and a float userargs parameter > # HAL ascribes no meaning to them > newm power-supply volt 1 0.1 > newm power-supply amps > newm power-supply mains 1 > newm power-supply fuse-ok 1 > > ---------------------------- > > Great, so what does this gain us? Nothing except we now have a named HAL > object 'power-supply' which has several member signals. It's a group > definition, nothing else - HAL per se will not do anything with it. We added > some int and float params to group and members. Not much more. > > That's why the support code in HAL and halcmd for dealing with groups and > members is fairly minimal. This turns us to > > How is a group used? > -------------------- > There are two users of the HAL group code: > > - halcmd - to define a group and members, as outlined above. > - one or several HAL components: to actually do something useful with a group: > > the HAL library would provide a few new functions: > - iterate of the list of groups, retrieve a group name and its parameters > - likewise, iterate over members of a group, and retrieve their names and > parameters. > > Now let's assume we would design a 'status reporter' for our power supply. > What that would do is inspect a HAL group periodically, and send a status > update message to some interested entity, maybe a GUI. > > That reporter could well be a userland HAL component. The outline would look > like so: > > - normal HAL initialisation. Nothing new here. > - ask HAL for group 'power-supply', retrieving group parameters. > - walk the list of group members, retrieving each member parameter. > - do something useful periodically, based on the above, running until > shutdown. > > at this point, the report component can ascribe some meaning to group and > member parameters. For instance, report could: > > - interpret group parameter 1 (1000) as a periodic update timer - meaning a > status update message should be sent unconditionally every second. > - interpret group parameter 2 (10) as a an internal change detection timer - > meaning members are checked as follows: > > -- default member parameter - zero - means no periodic check of this member > signal > -- member parameter '1' means - include in periodic check > ---- in the case of the float 'volt' value, the float param '0.1' is used as > an epsilon value to trigger the 'this value has changed' condition - float > equality comparisons are notoriously unreliable and an epsilon parameter > deals with that issue. > > Note that the behavior of the report component - timers, what to check for > etc - is completely driven by the HAL group and member attributes, so there > is actually very few code which is specific to the 'power supply' theme, in > fact its completely generic. > > The behavior would be as follows: > - every second, a status update is sent no matter what. > - if any of: mains goes away, the fuse is blown, or the voltage changes by > more than 0.1 volts a status update is sent after the next 10msec internal > poll interval. > > > Also, note a single component can report for an arbitrary number of groups > 'in one go'. > > The only thing left here is 'what happens when a change is detected'? > > Well, a HAL status update message is sent, by a mechanism which I already > discussed - using zeroMQ, and - for a start - the PUBLISH/SUBSCRIBE mechanism > to any interested readers - for instance HAL widgets in a GUI. > > I leave the details how such an update message is sent with ZeroMQ and > protobufs for a separate installment of the 'How to break LinuxCNC > effectively' miniseries; but be it enough here that even at that level, the > group mechanism is very useful: > > - the member list drives the encoding of values in the protobuf > serialisation. The message format need not know about the power supply > fields, types and so on - see requirement #1. > - the group name can be used as a ZeroMQ channel name - interested consumers > subscribe to the 'power-supply' ZMQ channel, and get updates - done. > > In summary: > > HAL groups and members support the definition of messages, with a late > binding approach, and with a fairly minimal extension to HAL per se, and no > interpretation or using code within the HAL library whatsoever. Any using > code will be components which are group/member aware. > > > Great. Now how's that going to help with dumping NML and replacing EMC_STAT > with this grandiose scheme? > ------------------------------------------------------------------------------------------------------- > > Well, really not much different like the above group definition, just longer. > All the needed types are there. > > Nested structures can be dealt with by HAL naming - for instance > "status.traj.probe_tripped" conveys the 'object nesting', even if its just a > HAL name in a flat namespace. > > The only thing which is not very well covered is arrays, for instance the > array of axes, or array of analog/digital I/O pins used in the interpreter. > That has it's reason in the fact that HAL doesnt support an array notation. I > dont see that as a reason to add one - that doesnt bring enough upside IMO. > The issue can be dealt by rolling the index into the HAL name - for instance, > 'status.digital-in-04' and thats' not different from current practice. > > So the above lets us define the equivalent of EMC_STAT with HAL groups. > > > So how's the migration of motion and iocontrol going to happen? > --------------------------------------------------------------- > > Unfortunately, motion and iocontrol dont have a clue about groups and members > right now, so how's that going to work? > > The answer to this is: they dont need to know zilch about groups or members, > that 'need to know' is confined to the reporting mechanism. > > What is needed, however, is: > > --> each scalar which is to be reported through the group mechanism needs to > be exported as a HAL pin so it can be linked to a member signal <---- > > Note that many members of emcmot_status_t are already exported as pin, so > there's nothing new here, and also the breakage potential is limited: the old > scheme can remain in place, and run in fact in parallel, while the new needed > pins are added and status reporting is defined. Eventually we can put > EMC_STAT, and emcmot_status_t to rest on the old code burial grounds. > > --- > > Please let me know if where above makes any sense, or not at all, where it is > unclear, and point me to out outright errors and omissions. > > - Michael > > > ps: for the very curious: I did a proof-of-concept of groups and members, > hal_lib and halcmd support, as well as a generic reporter with ZMQ messaging > and protobufs here: > http://git.mah.priv.at/gitweb?p=emc2-dev.git;a=tree;f=src/hal/halreport;hb=4259166865c681017b6009f041ac4d1ed5e16439 > > This is not anything mergeable, and way too bloated - it supports a group > notification mechanism from kernel modules which isnt needed, and lots of > other fluff which will got away. But it gives a rough effort indication: > v0.001 of the generic HAL reporter component is less than 1000 lines; the > hal_lib.c group/member support much less than that. > > A very basic Python subscriber to the reporter's channel is here: > http://git.mah.priv.at/gitweb?p=emc2-dev.git;a=blob;f=src/hal/halreport/sub.py;h=85ffeae0e4a576755aaab186d7378b29cdeca75d;hb=4259166865c681017b6009f041ac4d1ed5e16439 > > Some things I see missing: 1 -- I would still like to have arrays of characters (strings), and I'd like them supported up to the gcode (interpreter) level. I should be able to call a (gcode) function that engraves the value of a string parameter (my name). 2 -- Can a group be a member of another group? If not, I think that is a limitation. Today, we have structures that are contained within other structures (although they must be compiled in). 3 -- To simplify, my primitive elements would be: characters, integers, reals, bools. My compound elements would be groups (of primitives or compounds) and arrays (of primitives or compounds). Arrays would have variable (possibly with a maximum specified size -- although I prefer not) length. Personally, I would take this opportunity to change to a more standard external data representation. JSON comes to mind for an external (human readable) representation. See <http://www.json.org/> for a definition of the representation (in a single page) and links to a flock of libraries in many programming languages. Again, thank you Michael, for your leadership in this area. Regards, Ken > > > > > > > ------------------------------------------------------------------------------ > Free Next-Gen Firewall Hardware Offer > Buy your Sophos next-gen firewall before the end March 2013 > and get the hardware for free! Learn more. > http://p.sf.net/sfu/sophos-d2d-feb > _______________________________________________ > Emc-developers mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/emc-developers ------------------------------------------------------------------------------ Free Next-Gen Firewall Hardware Offer Buy your Sophos next-gen firewall before the end March 2013 and get the hardware for free! Learn more. http://p.sf.net/sfu/sophos-d2d-feb _______________________________________________ Emc-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/emc-developers
