Hi Lukasz,
great news, even if I have to admit I didn't quite understand most of
the problems you were having. But it's great to hear that you've made
progress :)
I think in time we might even get rid of Netty in total. If we want to
go more and more towards fully generated drivers, we need a core in all
of our languages that we can work with similarly. Netty is currently
Java only (even if there's a DotNetty for C#).
Looking forward to finally finishing some of the Firmware-Building stuff
that was consuming almost all of my time recently and I hope I can then
help you and the others with stuff they might be needing help for :-)
Chris
On 21.07.21 00:32, Łukasz Dywicki wrote:
> I am bringing back CAN stuff for PLC4J since ApacheCon Asia had forced
> me to finally move forward. Code will follow shortly as I am reaching
> the stabilization point.
>
> A quick summary of changes.
> 1) I brought a `plc4j-transport-can` which contains a `CANTransport`.
> Main role of this module is definition of CAN related APIs such as
> `CANFrameBuilder`, `FrameData` which can be used by drivers who intend
> to interact with CAN bus.
> 2) SocketCAN transport now uses generated mspec structure to map binary
> form of blob we receive from JavaCAN library. By this way, at least some
> portion of mapping is portable across languages.
> 3) CANOpen mspec defines its own frame type which is used to implement
> entire protocol logic. This type has no common denominator with CAN or
> socketcan frame. It represents only information specific to this
> particular protocol.
> 4) Earlier work which been done with Sebastian and custom stacks opened
> drivers to do a little bit more than just declare types. This allows
> CANopen driver to inject a wire adapter from own message definition to
> the desired format.
>
> I had a fairy long battle with compiler and myself trying to abstract
> things away and have a top level definition of CANFrame as well as
> CANOpenFrame. In the end I failed making it flying, despite of small
> successes either in transport or protocol itself.
> Given that we use a lot of code generation which is unaware of language
> specific syntax or features I ended up leaving these as-is.
> A main reason why such approach didn't go very well was amount of
> complications it created. We had then:
> 1) CANFrame
> 2) SocketCANFrame (generated)
> 3) SocketCANFrameWrapper extends SocketCANFrame implements CANFrame
> 4) CANTransport had a #CANFrameBuilder interface to hide it from driver
> 5) CANOpenFrame (generated)
> 6) Builder which could in theory bridge above type to CANFrame
>
> As you can see, a lot of troubles. In order to keep whole thing straight
> I left below:
> 1) FrameData
> 2) SocketCANFrame (generated)
> 3) CANOpenFrame (generated)
> 4) CANTransport declares few additional indicators:
> a) Wire type - SocketCANFrame and associated MessageIO
> b) It must return adapter from wire type to FrameData
> c) It must provide frame builder back to wire type
> 5) There is an additional `CANDriverBase`.
>
> By this way we are able to utilize generated code as well as use any of
> further APIs coming from third parties to interact with vendor specific
> transports or APIs. Most importantly, as long as transport can declare
> wire encoding and provide it to driver, it will work.
>
> Few words about motivations.
> Given that CANOpen does rely on own processing logic such as serviceId
> and nodeId which are delivered from CAN frame identifier (COB id) any
> unification attempt around CANFrame lead to massive amount of changes in
> existing protocol logic. I did not want to pollute that logic with bit
> masks, int/short casts and so on. Additionaly I still have in back of my
> mind compatibility/portability of logic to CAN 2.0B/CAN FD.
> I simply had to look for other way.
> Since we have separate root frame for CANopen and transport we can do
> better tests of protocol implementation, without introducing a transport
> specific encoding.
> With CANFrame as an universal wire format for CANopen protocol I was
> forced to do a comparison of serialized byte arrays instead of CANopen
> objects. Even if it would fly somehow with CAN 2.0a (8 byte payloads), I
> could not imagine this working for CAN FD (payloads up to 64 bytes).
> Loosing a test visibility, especially after improvements test framework
> received recently, was not acceptable.
>
> Above approach with asymmetric API with separate wire format to write to
> bus and frame data to receive from bus allows to retain a single
> transformation logic from any transport. It works even if we don't have
> a canonical form of a CAN frame.
> Main place of work is done in a portable translation layer with a new
> type called `FrameHandler`. This translation layer is responsible for
> two call paths:
> 1) wire -> FrameData -> driver frame handler -> driver frame
> 2) driver frame -> driver frame handler+transport frame builder -> wire
>
> As you can see it uses extensively CAN transport information in
> combination with FrameHandler implemented by an driver.
> I think that over time we can seek additional optimizations or
> simplifications in above design.
>
> For any new CAN driver independent of wire format it is necessary to:
> 1) do an mspec
> 2) extend `Plc4xCANProtocolBase` (it makes #decode method public)
> 3) provide a `FrameHandler` implementation specific to driver
> 4) wire everything together using custom stack builder
>
> The end result of all gymnastics I made is this:
> customStack.builder(trans.getMessageType(), trans.getMessageIO)
> .withProtocol(cfg-> {
> CANOpenProtocolLogic protocolLogic = new CANOpenProtocolLogic();
> ConfigurationFactory.configure(cfg, protocolLogic);
> return new CANDriverAdapter<Message, CANOpenFrame>(
> protocolLogic,
> trans.getMessageType(),
> trans.adapter(),
> new CANOpenFrameDataHandler(trans.getTransportFrameBuilder())
> );
> })
> .withDriverContext(cfg -> new CANOpenDriverContext())
> .withPacketSizeEstimator(cfg -> trans.getEstimator())
> .littleEndian()
> .build();
>
> For now the CANDriverAdapter is very close to netty apis we have. Not in
> terms of what it does, but in masking the sendToWire methods.
> It was necessary to inject additional translation logic implemented by
> driver frame handler. It is a bit unfortunate that there is no better
> way, especially that ConversationContext and SendRequestContext have own
> quirks which may cause additional maintenance cost in there.
>
> With all this I guess we might be able to see if this is portable to
> other drivers, especially ones which do rely on raw ethernet! :)
>
> Best,
> Łukasz
>
>
> On 14.04.2021 02:36, Łukasz Dywicki wrote:
>> I began to scratch surface of Driver/Protocol/Context and here is what I
>> found.
>>
>> Currently it is *impossible* to do any customization with
>> ProtocolStackConfigurer or GeneratedDriverBase without really copying
>> whole classes.
>> Trouble I see is:
>> 1) Protocol stack configurer rely on classes instead of actual instances.
>> Is there any reason why packet estimator and byte discarder are passed
>> there as classes and not actual instances? These are functions which
>> *should* not have side effects hence we should not require an reflection
>> to create instance of these.
>> 2) Driver which is asked to create protocol stack has *no access* to
>> actual configuration.
>> 3) Transport is "lost in transition" while creating pipelines.
>> It is not retained nor accessible to stack configurer making it
>> impossible to ie. make transport dependent stack configuration. As much
>> as this situation is not very common. yet we either force user to pass
>> extra URL parameters or let drivers do their defaults based on transport.
>>
>> Very simplified startup flow for driver is:
>>
>> generated driver initialization ->
>> create configuration
>> find transport ->
>> configure transport
>> transport -> create channel factory
>> create connection ->
>> create field handler
>> create value handler
>> create stack configurer
>>
>> My proposal is to switch it to:
>> create configuration
>> find transport ->
>> configure transport
>> transport -> create channel factory
>> create connection ->
>> create field handler
>> create value handler
>> create stack configurer*(configuration, transport)*
>>
>> Another finding I have - estimators and discarders are created by
>> reflection in hope to receive configuration from URI parameters.
>> Such situation so far does not happen and I think its rather rare (yet
>> we can face it some day), thus I would opt for having configuration
>> calls with them, however I would move that part from stack configurer
>> above - to the driver. In case if driver expects configuration options
>> for its wire processors it must enable it explicitly.
>> This will also leave a space to create base class for CAN drivers and
>> utilize transport and configuration to create estimator and discarder
>> dynamically.
>>
>> Looking forward to hear your opinion about above changes.
>>
>> Best,
>> Łukasz
>>
>> On 29.03.2021 06:43, Andreas Oswald wrote:
>>> Hi Łukasz,, hi everybody,
>>>
>>> that sounds pretty interesting, do you have plans to to implement "higher
>>> level" protocols based on CAN e.g. UDS? Maybe we should exchange ideas as
>>> my daytime job ist about developing a software package that takes care of
>>> Software Versions and parameter sets for electrical vans, where we rely
>>> heavily on UDS. Unfortunately not all of our ECUs implement fine UDS.
>>>
>>> When Chris introduced me to the project I guess a bit more than two years
>>> ago, Slack was the medium to use.
>>>
>>> Maybe someone can give me a hint about how communication is dealt with.
>>>
>>> Good to be back 😉
>>>
>>> Have a great start for the new week.
>>>
>>> Take care
>>>
>>> Andreas
>>>
>>> Andreas Oswald
>>> Marienstraße 12
>>> 52223 Stolberg
>>> Germany
>>>
>>> ++49 2402 905 3852
>>> ++49 172 426 1598 (also Signal)
>>> WhatsApp/Telegram on ++49 157/38459358
>>>
>>> ________________________________
>>> Von: Łukasz Dywicki <[email protected]>
>>> Gesendet: Montag, 29. März 2021 02:18
>>> An: [email protected] <[email protected]>
>>> Betreff: Further updates to CAN stuff in plc4j
>>>
>>> Hey all,
>>> I began to think how to cover basic interactions with CAN bus which
>>> could serve a custom systems or ones which are closer to some narrow
>>> cases. My intention here is to offer simplest possible way to become
>>> CANbus client without any specific application layer protocol.
>>> Results of my thought process are following.
>>>
>>> Driver operations: CAN as protocol is rather event based hence CAN
>>> driver should not support "read" but only write and subscribe. A "read"
>>> in our api indicate rather an synchronous call for which we expect an
>>> answer. This is not a case with CAN where you can keep broadcasting but
>>> never expect any answer back or could have multiple replies. While
>>> downstream protocols can offer such functionality (canopen does) at most
>>> basic level we can't provide any handling for that. Definitely it would
>>> make hard to port driver over different languages.
>>> If this driver gets implemented end users can use it as a very thin
>>> client API which gives them access to the bus and covers very basic
>>> handling of subscriptions for specific COBs (CAN Object Identifier) and
>>> possibly basic IEC types via field handlers.
>>>
>>> Transports: an awful part of CAN is amount of APIs which can be used to
>>> access a bus. The low level stuff is done by microcontrollers and hidden
>>> from applications who interact with a bus. In the end socketcan, slcan,
>>> usb2can or whathever-to-can are just multiple ways to publish CAN frame
>>> onto physical medium.
>>> I did hang on this for a moment, cause driver syntax we have is briefly:
>>> <protocol>:<transport>://<transport-options>?<driver-options>
>>>
>>> Currently connection string for CANopen looks like this:
>>> canopen:socketcan://vcan0?heartbeat=false
>>> Yet for slcan or can2usb it could look like this:
>>> canopen:slcan://dev/ttyUSB0;baud=50000?heartbeat=false
>>> canopen:can2usb://COM4;baud=50000?heartbeat=false
>>> (eventually with baud= parameter in query string).
>>>
>>> Whatever syntax we use to express transport and its configuration the
>>> problem is later shifted to the edge of Driver and Transport. Different
>>> APIs might bring different frame length estimators or discarded byte
>>> handling nor filtering capabilities. Currently driver is one who is
>>> responsible for that, yet in case of CAN - socketcan uses 0x00 to always
>>> pad data while serial bridges do not require it at all. At present I see
>>> no way to fit it without breaking some things.
>>> CAN transport essentially must provide above estimators. Maybe in future
>>> some other features specific to this transprot ie. message filtering
>>> which seems to be very common thing. For driver it must provide a way to
>>> publish "COB + byte[]" and receive "COB + byte[]".
>>>
>>> So far we have support only for socketcan transport. It did the job to
>>> get into CAN, yet it is not the only one. Its not portable over all
>>> platforms. Because of above condition canopen driver actually depends on
>>> socketcan transport and its encoding of frames.
>>> My long term plan in this area is extraction of CANFrameBuilder part out
>>> of canopen driver to plc4j-transprot-can. This transport should define
>>> CAN specific transport APIs. What do you think about that?
>>>
>>> Once this step is made we can bring simplistic CAN driver with very low
>>> effort. It will also open a path for other CAN transports which could
>>> utilize yet another hardware and serve other operating systems.
>>>
>>> Best,
>>> Łukasz
>>>
