Re: [DISCUSS] How to do discovery?
I can share some of my findings I had with Siemens, Beckhoff and other stuff which is generally speaking.. easier to discover. ;-) Long story short - most of tools has some ways to find them over network. For Siemens it is profinet-dcp or lldp. Both of these are packed into ethernet frames. I believe that lldp has different responsibilities, than dcp, ie. you can use lldp to make plc blink with its diodes. Beckhoff device discovery. from what I have found, is based on udp packets which are described nowhere. You need to send first a broadcast in order to get a reply which gives you back device name and its ip which is sufficient to setup ADS routes. And that's the part where it gets tricky - route setup requires username/password and in case of new devices - could require a certificate. This username / password is necessary to get ADS link working. Without that you can still make TCP connection but your packets go straight to C:/dev/null. ;-) To give you my feeling on this one - its gonna be pretty hard to have discovery embedded in drivers cause this process is different than making connection. I'm not saying that its completely unrelated, but I think it deserve its own "driver free" SPI. For consumer stuff there is UPNP and mDNS which you can use, but not in here. :/ Discovery process is actually composed from following steps: - Preparing discovery - Try locating end devices - Construct connection templates - Provide information about required params (ie. user+pw) - Let caller provide missing parameters, if any - Launch actual driver connection I've implemented beckhoff discovery with openhab binding here: https://github.com/ConnectorIO/connectorio-addons/tree/master/binding/plc4x/beckhoff/src/main/java/org/connectorio/binding/plc4x/beckhoff/internal/discovery/udp Code is awful, due to manual packet parsing, partially because of OH discovery API design. It is parameter-less, meaning that you need to scan all network interfaces, then make broadcasts over them and listen for answers. The interface in case of ADS is important cause you need to send own IP address to PLC in order to setup route. For DCP is kinda similar - you need to get interface name first or get over all of them. Cheers, Łukasz On 08.07.2020 18:59, Christofer Dutz wrote: > Ok ... picking up the ball on this again :-) > > So yesterday I added 4 empty interfaces to the SPI. All are in the > "org.apache.plc4x.java.spi.discovery" package. > I guess I could probably omit the "SupportsDiscovery" interface which all > others extend, but I'll leave it for now. > > In general there seem to be 3 types of discovery mechanisms (regarding > network technology): > 1) Actively connect to a device > 2) Passively listen to traffic > 3) Send some broadcast packets (Not 100% sure about the name) > > If I missed one, please tell us about it. > > I guess the broadcast has to be differentiated a little as I have seen > broadcasts events being sent out and the other members respond by a broadcast > themselves or they send a direct packet to the source of the broadcast. > > I also added a "discovery" driver in the sandbox, but that's just a work > proposal for now ... if we decide to do it that way, if we want to extend the > PLC4X API directly or even do both, I guess that's up to us. For now I'd like > to concentrate on getting first discoveries working somehow. > > So my first steps would be to propose some methods in the corresponding > Discovery interfaces. As soon as that's done I would take a driver and try to > implement the discovery for that. > > So ... my friends ... I really need your input on this ... I do know some of > you already have thought about discovery and I would like to channel these > thoughts into something productive :-) > > > Looking forward to some feedback. > > Chris > > > > > Am 01.07.20, 16:00 schrieb "Christofer Dutz" : > > Hi all, > > Thanks Otto for that link ... looks interesting, even if this might be > more interesting for the internals of how we find things. > > In general the discovery will be 3 things: > 1) Detecting there is something at a given IP/Port > 2) Detecting the protocol > 3) Getting information about the thing using the protocol we detected > > Active Scenario: > > - So for example using TCP/UDP to detect there is something on IP > 10.10.64.20 (Could be a ping) > - Next to knock on typical ports depending on the protocols (possibly > scanning ports) > - So as soon as we know on IP 10.10.64.20 there is something and it's got > an open port on port 102 > - So now the S7 protocol could raise an arm and ask to give it a try as > it knows it usually operates on port 102 > - It would try to establish a connection and here could find out the type > of PLC using the old S7 protocol > So in the end we'd know on which device a given protocol is supported and > possibly even which type of PLC it is and which software
Re: [DISCUSS] How to do discovery?
Ok ... picking up the ball on this again :-) So yesterday I added 4 empty interfaces to the SPI. All are in the "org.apache.plc4x.java.spi.discovery" package. I guess I could probably omit the "SupportsDiscovery" interface which all others extend, but I'll leave it for now. In general there seem to be 3 types of discovery mechanisms (regarding network technology): 1) Actively connect to a device 2) Passively listen to traffic 3) Send some broadcast packets (Not 100% sure about the name) If I missed one, please tell us about it. I guess the broadcast has to be differentiated a little as I have seen broadcasts events being sent out and the other members respond by a broadcast themselves or they send a direct packet to the source of the broadcast. I also added a "discovery" driver in the sandbox, but that's just a work proposal for now ... if we decide to do it that way, if we want to extend the PLC4X API directly or even do both, I guess that's up to us. For now I'd like to concentrate on getting first discoveries working somehow. So my first steps would be to propose some methods in the corresponding Discovery interfaces. As soon as that's done I would take a driver and try to implement the discovery for that. So ... my friends ... I really need your input on this ... I do know some of you already have thought about discovery and I would like to channel these thoughts into something productive :-) Looking forward to some feedback. Chris Am 01.07.20, 16:00 schrieb "Christofer Dutz" : Hi all, Thanks Otto for that link ... looks interesting, even if this might be more interesting for the internals of how we find things. In general the discovery will be 3 things: 1) Detecting there is something at a given IP/Port 2) Detecting the protocol 3) Getting information about the thing using the protocol we detected Active Scenario: - So for example using TCP/UDP to detect there is something on IP 10.10.64.20 (Could be a ping) - Next to knock on typical ports depending on the protocols (possibly scanning ports) - So as soon as we know on IP 10.10.64.20 there is something and it's got an open port on port 102 - So now the S7 protocol could raise an arm and ask to give it a try as it knows it usually operates on port 102 - It would try to establish a connection and here could find out the type of PLC using the old S7 protocol So in the end we'd know on which device a given protocol is supported and possibly even which type of PLC it is and which software version it is (Does it support S7plus?) Passive scenario: - We start capturing data - A packet is received - Each driver is asked "Here's a tcp packet on port 102 you wanna check it out?" - If the driver says "sure" because he knows the port or simply wants to try to decode every packet, he'll start to decode the packet - If it fails, well then it's probably not that protocol - If it succeeds it will give a positive response possibly containing additional information Unless we happen to intercept a packet where the PLC provides information about itself we can't actively ask for more information Possibly a combination of passive detection of thing and protocol with additional active interrogations would be a good thing to implement. Scanning an IP range and ports is potentially problematic as this does put quite some stress on unhardened devices and will probably let some alarm-bells ring. We could make plc4x limit this scanning to intervals so perhaps such a scan is only done in the night and only once a week on the weekend. Chris Am 01.07.20, 13:55 schrieb "Otto Fowler" : https://old.zeek.org/development/howtos/dpd.html On June 30, 2020 at 13:32:57, Christofer Dutz (christofer.d...@c-ware.de) wrote: Just had another idea ... How about giving this driver no real transport at all (think we have the dummy transport ... that works like a charm) and then to provide the means in the subscription addresses? This way I could start discovering serial, passive and active ... and perhaps even multiple instances with just one connection ... Chris Am 30.06.20, 19:09 schrieb "Christofer Dutz" : Hi folks, for the past weeks I have been thinking about how we could approach the “discovery” topic. I think I just had an idea and would like some feedback from you. So I was thinking about how we could model such a discovery API. In general it would look pretty different to the existing APIs … at least I thought. The Idea I just had, was: How about we create a new “discovery” driver? This can use both the serial transport as well as the passive-mode transport or even tcp transport for protocols that allow active discovery mechanisms.
Re: [DISCUSS] How to do discovery?
Hi all, Thanks Otto for that link ... looks interesting, even if this might be more interesting for the internals of how we find things. In general the discovery will be 3 things: 1) Detecting there is something at a given IP/Port 2) Detecting the protocol 3) Getting information about the thing using the protocol we detected Active Scenario: - So for example using TCP/UDP to detect there is something on IP 10.10.64.20 (Could be a ping) - Next to knock on typical ports depending on the protocols (possibly scanning ports) - So as soon as we know on IP 10.10.64.20 there is something and it's got an open port on port 102 - So now the S7 protocol could raise an arm and ask to give it a try as it knows it usually operates on port 102 - It would try to establish a connection and here could find out the type of PLC using the old S7 protocol So in the end we'd know on which device a given protocol is supported and possibly even which type of PLC it is and which software version it is (Does it support S7plus?) Passive scenario: - We start capturing data - A packet is received - Each driver is asked "Here's a tcp packet on port 102 you wanna check it out?" - If the driver says "sure" because he knows the port or simply wants to try to decode every packet, he'll start to decode the packet - If it fails, well then it's probably not that protocol - If it succeeds it will give a positive response possibly containing additional information Unless we happen to intercept a packet where the PLC provides information about itself we can't actively ask for more information Possibly a combination of passive detection of thing and protocol with additional active interrogations would be a good thing to implement. Scanning an IP range and ports is potentially problematic as this does put quite some stress on unhardened devices and will probably let some alarm-bells ring. We could make plc4x limit this scanning to intervals so perhaps such a scan is only done in the night and only once a week on the weekend. Chris Am 01.07.20, 13:55 schrieb "Otto Fowler" : https://old.zeek.org/development/howtos/dpd.html On June 30, 2020 at 13:32:57, Christofer Dutz (christofer.d...@c-ware.de) wrote: Just had another idea ... How about giving this driver no real transport at all (think we have the dummy transport ... that works like a charm) and then to provide the means in the subscription addresses? This way I could start discovering serial, passive and active ... and perhaps even multiple instances with just one connection ... Chris Am 30.06.20, 19:09 schrieb "Christofer Dutz" : Hi folks, for the past weeks I have been thinking about how we could approach the “discovery” topic. I think I just had an idea and would like some feedback from you. So I was thinking about how we could model such a discovery API. In general it would look pretty different to the existing APIs … at least I thought. The Idea I just had, was: How about we create a new “discovery” driver? This can use both the serial transport as well as the passive-mode transport or even tcp transport for protocols that allow active discovery mechanisms. So you would simply create a “connection” to something like “discovery:raw//eh1” or “discovery:serial:///dev/ttyS0” … now this driver would be a little special. It would internally query the list of drivers available on the given system, the same way the DriverManager does it. But it would check each driver if it implements an interface “SupportsDiscovery” (or whatever name we give it) … So it would then initialize an instance of all drivers supporting discovery. So in the end the DiscoveryDriver would simply try to feed each packet to each of the drivers and have them check if they can make sense out of that. If they do, they would start emitting events just the same way a resource subscription does. (Of course we should probably apply some filtering mechanism to avoid too much overload) So a client wanting to use discovery, would use the normal PLC4X API to connect and then would simply subscribe to the datastream produced by that. So in the end we wouldn’t be changing anything with the user-facing API and all could be done internally … and the cool thing we would get all the integrations working with this without modifications for free :-) … so you could start simply streaming the discovery data to StreamPipes or Kafka or log it in IoTDB for intrusion detection or other crazy stuff What do you think? I have to admit I am currently absolutely happy with this idea … so please … bombs away … tear it apart ;-) Chris
Re: [DISCUSS] How to do discovery?
https://old.zeek.org/development/howtos/dpd.html On June 30, 2020 at 13:32:57, Christofer Dutz (christofer.d...@c-ware.de) wrote: Just had another idea ... How about giving this driver no real transport at all (think we have the dummy transport ... that works like a charm) and then to provide the means in the subscription addresses? This way I could start discovering serial, passive and active ... and perhaps even multiple instances with just one connection ... Chris Am 30.06.20, 19:09 schrieb "Christofer Dutz" : Hi folks, for the past weeks I have been thinking about how we could approach the “discovery” topic. I think I just had an idea and would like some feedback from you. So I was thinking about how we could model such a discovery API. In general it would look pretty different to the existing APIs … at least I thought. The Idea I just had, was: How about we create a new “discovery” driver? This can use both the serial transport as well as the passive-mode transport or even tcp transport for protocols that allow active discovery mechanisms. So you would simply create a “connection” to something like “discovery:raw//eh1” or “discovery:serial:///dev/ttyS0” … now this driver would be a little special. It would internally query the list of drivers available on the given system, the same way the DriverManager does it. But it would check each driver if it implements an interface “SupportsDiscovery” (or whatever name we give it) … So it would then initialize an instance of all drivers supporting discovery. So in the end the DiscoveryDriver would simply try to feed each packet to each of the drivers and have them check if they can make sense out of that. If they do, they would start emitting events just the same way a resource subscription does. (Of course we should probably apply some filtering mechanism to avoid too much overload) So a client wanting to use discovery, would use the normal PLC4X API to connect and then would simply subscribe to the datastream produced by that. So in the end we wouldn’t be changing anything with the user-facing API and all could be done internally … and the cool thing we would get all the integrations working with this without modifications for free :-) … so you could start simply streaming the discovery data to StreamPipes or Kafka or log it in IoTDB for intrusion detection or other crazy stuff What do you think? I have to admit I am currently absolutely happy with this idea … so please … bombs away … tear it apart ;-) Chris
Re: [DISCUSS] How to do discovery?
Just had another idea ... How about giving this driver no real transport at all (think we have the dummy transport ... that works like a charm) and then to provide the means in the subscription addresses? This way I could start discovering serial, passive and active ... and perhaps even multiple instances with just one connection ... Chris Am 30.06.20, 19:09 schrieb "Christofer Dutz" : Hi folks, for the past weeks I have been thinking about how we could approach the “discovery” topic. I think I just had an idea and would like some feedback from you. So I was thinking about how we could model such a discovery API. In general it would look pretty different to the existing APIs … at least I thought. The Idea I just had, was: How about we create a new “discovery” driver? This can use both the serial transport as well as the passive-mode transport or even tcp transport for protocols that allow active discovery mechanisms. So you would simply create a “connection” to something like “discovery:raw//eh1” or “discovery:serial:///dev/ttyS0” … now this driver would be a little special. It would internally query the list of drivers available on the given system, the same way the DriverManager does it. But it would check each driver if it implements an interface “SupportsDiscovery” (or whatever name we give it) … So it would then initialize an instance of all drivers supporting discovery. So in the end the DiscoveryDriver would simply try to feed each packet to each of the drivers and have them check if they can make sense out of that. If they do, they would start emitting events just the same way a resource subscription does. (Of course we should probably apply some filtering mechanism to avoid too much overload) So a client wanting to use discovery, would use the normal PLC4X API to connect and then would simply subscribe to the datastream produced by that. So in the end we wouldn’t be changing anything with the user-facing API and all could be done internally … and the cool thing we would get all the integrations working with this without modifications for free :-) … so you could start simply streaming the discovery data to StreamPipes or Kafka or log it in IoTDB for intrusion detection or other crazy stuff What do you think? I have to admit I am currently absolutely happy with this idea … so please … bombs away … tear it apart ;-) Chris
