Hi Julian, Indeed, your concerns are valid regarding creating an already abandoned TLP.
In that case, I would opt for a private repo on GitHub which we invite everyone willing to help work on it and as soon as we have something workable, we make it public as part of a TLP? This avoids at least the fear of creating a new TLP that’s product doesn’t even see the light of day. And as soon as we have something we want to show the world, we join Apache? Would that be a compromise? Chris From: Julian Feinauer <[email protected]> Date: Monday, 28. November 2022 at 13:14 To: [email protected] <[email protected]> Subject: AW: Building a PLC4X and IoTDB Historian? Hey, as I said.. we could just do it as part of one of the TLPs like PLC4X or IoTDB easily or in a private repo as well, for sure. I just want to avoid to create a TLP which is soon abandoned, that’s my only concern. Regarding all your comments I totally agree and am with you. Julian Von: Christofer Dutz <[email protected]> Datum: Montag, 28. November 2022 um 13:07 An: [email protected] <[email protected]> Betreff: Re: Building a PLC4X and IoTDB Historian? Hi Julian, well … I would be hesitant to try something like this without the legal shield of the ASF … from what I’ve learned in the early days, Siemens and Co. continuously threatened to sue me for my work on PLC4X. The only thing that helped me then, was to say: I am doing everything legally, so you’d have trouble succeeding with that and you also can’t sue me but would have to sue the ASF and good luck on that not backfiring badly on you publicly. I know that in the sector of drivers in the past there were several occasions where open-source developers were annoyed so much by nuisance law-suites, that they gave up. I also know Historians are silly expensive systems. I haven’t seen one under 100k€ and the annual fees for certified solutions tend to start at 100k€/year/production-line … so there’s a lot of money for them to fear losing. So yes: I’m too scared to try something like this publicly without a legal shield. We could however work on something like this in a protected repo at GitHub etc. and go to TLP as soon as we have something. Would that be an option? Chris From: Julian Feinauer <[email protected]> Date: Monday, 28. November 2022 at 12:57 To: [email protected] <[email protected]> Subject: AW: Building a PLC4X and IoTDB Historian? Hey Chris, wrt testing I meant how „sustainable” the community is, that we intend to building. You know the effect pretty well if you ask “How wants to start this” everybody is like “me, me, me” and after 2 weeks you stand there alone. And one important aspect of a TLP for me is that the community is also “resilient” and will last “long” (whatever that means). And this is something I think is hard to guarantee in the current state. Technically I have no worries, this is something we would get to work rather easily, I see no “magics” hidden there. Julian Von: Christofer Dutz <[email protected]> Datum: Montag, 28. November 2022 um 12:48 An: [email protected] <[email protected]> Betreff: Re: Building a PLC4X and IoTDB Historian? Hi Julian, Well, the problem with starting at one Project as a subproject … We then need to make everyone involved or wanting to be involved in the other project to become committers in the other. I wouldn’t have problems with doing it under the hood of IoTDB, personally. A separate Project would have the benefit of giving the industry an impression that this is something built just for them instead of being a side-product of something that’s already there (I have given up expecting the Automation industry to use any form of common sense ;-) … guess that will be my definition of Industry 5.0 … “connected everything + common sense”) The incubator is intended for teaching the initial community how things work at Apache, I guess as we’d be starting with mainly people that have already brought projects from outside into the incubator and into TLPs, this teaching effort will not exist in this case. Even Justin mentioned that this would probably be a case for straight TLP. Regarding the testing … not sure I understand what you want to test? The idea of an open-source historian? I mean the need exists … I’ve seen it in multiple occasions and our friends from IoTDB have confirmed they have built or at least seen IoTDB serve as such a system. Not 100% sure what you want to try out. But I agree … we should probably start with an (online) workshop, as this time we’d probably be spread out a bit wider geographically, than with our PLC4X workshops in the past. Chris From: Julian Feinauer <[email protected]> Date: Monday, 28. November 2022 at 12:37 To: [email protected] <[email protected]> Subject: AW: Building a PLC4X and IoTDB Historian? Hey, I understand your point and also agree generally. But I personally dislike the straight to TLP route as this is something I’d like to try out first. And my point with the separate repo (or whatever) was just to make the community ready for the Incubator (where a codebase and community normally should exist). But the easiest way in any regard would be to start of as a subproject in an existing PMC. This would only require to get another repo and we could start (formally no need to even register it as a subproject). And the repo thing could be done by you alone I think as Chair. Next steps would then be more of brainstorming and workshoping and building something (whatever?). I think the timing is quite good before Christmas because if its intereting, I see myself contributing over the Holidays (better don’t show this mail to my wife…). Best Julian Von: Christofer Dutz <[email protected]> Datum: Montag, 28. November 2022 um 09:27 An: [email protected] <[email protected]> Cc: Tim Mitsch <[email protected]>, [email protected] <[email protected]> Betreff: Re: Building a PLC4X and IoTDB Historian? Hi Julian, happy you like the idea ;-) We would probably outclass the established ones in point of price ;-) Right now, I think we should probably set up something that bundles IoTDB and PLC4X and then the “code of the project” is the bubble-wrap, that brings the two together and then provides an API that is equivalent to existing Historian products. I think the only way we can score points here, is if switching to it is as easy as possible and that only works if it’s sort of a drop-in replacement. Currently also discussing with other board members and the IPMC chair … he at least would not see us in the incubator. If there are enough PMC members of other projects and Apache Members on board, we could take the straight to TLP route. And I would really like to keep it at Apache and not set up something outside. This for multiple reasons: * Protection (The market for the established products is huge and the companies behind them are also huge with huge legal departments, I would fear the same as I did with PLC4X, when I started it … don’t want to be stuck in nuisance lawsuits, just filed for keeping us occupied) * The satement: we want Open-Source to be accepted by the industry and for me “Apache” is the form of Open-Source that I admire most. If it’s built up out of Apache stuff, it should be an Apache project (If we establish this in a way, that the industry notices Apache as a “vendor”, this will benefit many other projects) Chris From: Julian Feinauer <[email protected]> Date: Monday, 28. November 2022 at 09:17 To: [email protected] <[email protected]> Cc: Tim Mitsch <[email protected]>, [email protected] <[email protected]> Subject: AW: Building a PLC4X and IoTDB Historian? Hi all, first of all.. sorry for coming in so late. I think this is a great idea and there really is a product market fit for such a solution. I think technology wise its not "that complicated" to build such a system, I think many of the details that are necessary for a good adoption of such a software is in the areas around like documentation, marketing, more marketing, even more and even bolder marketing and sales (how to sell an open source project??). But technology wise I really like the idea and I think with the technology one has today it is very easy to "outclass" these established systems e.g. in performance, efficiency and als user friendliness. The only thing which might be challenging for the future are things like plugins or an extension system to allow users to customize their installation. Regarding the "separate project" approach I am not 100% certain. Personally, I would consider starting EITHER as a subproject in an existing PMC (PLC4X?) or as a separate undertaking on GitHub or somewhere else and not directly go to the Incubator or something. Because I think we should really find out if there is enough developer interest to build and sustain such a system. But I'm totally in for such a system either way! Julian PS.: Also forwarding the email to Björn and Tim who work(ed) a lot with historians Von: Christofer Dutz <[email protected]> Datum: Montag, 28. November 2022 um 08:56 An: [email protected] <[email protected]> Betreff: Re: Building a PLC4X and IoTDB Historian? Hi Xiangdong, that’s perfect that some of you folks would be on board. I should probably check, if there are enough PLC4X folks on board too … would be a shame if it was just me ;-) And regarding the name … yeah … you are absolutely right: Historian might really be the ideal choice … as it instantly explains what it is … A Historian from the Apache Software Foundation and probably also no bias towards any regional culture. So (Speaking to my fellow PLC4X folks … who would be on board with this?) Chris From: Xiangdong Huang <[email protected]> Date: Monday, 28. November 2022 at 04:24 To: [email protected] <[email protected]> Subject: Re: Building a PLC4X and IoTDB Historian? > I guess I would sort of also vote for a separate Project. Would however only > make sense if some people from both of our projects would join in. Agree, I think the iotdb developers in Timecho can join. > And name-wise … one of the names of Historic Historians (Ideally a Greek one) > would make it into my top 10 ;-) … sort of like Hudson or Jenkins are > “famous” Buttler names ;-) If we consider famous names (well, there will be culture bias), I'd like to suggest this one: https://en.wikipedia.org/wiki/Sima_Qian , as "Records of the Grand Historian" is tooo famous (at least in China). BTW. naming "Apache Historian" directly may be another option. Best, ----------------------------------- Xiangdong Huang School of Software, Tsinghua University Christofer Dutz <[email protected]> 于2022年11月27日周日 22:56写道： > > How about Apache Ephorous? ;-) > > https://en.wikipedia.org/wiki/Ephorus > Ephorus of Cyme (/ˈɛfərəs/<https://en.wikipedia.org/wiki/Help:IPA/English>; > Greek<https://en.wikipedia.org/wiki/Greek_language>: Ἔφορος ὁ Κυμαῖος, > Ephoros ho Kymaios; c. 400 – 330 BC) was an ancient > Greek<https://en.wikipedia.org/wiki/Ancient_Greece> > historian<https://en.wikipedia.org/wiki/Historian> known for his universal > history<https://en.wikipedia.org/wiki/Universal_history>. > > Chris > > From: Christofer Dutz <[email protected]> > Date: Sunday, 27. November 2022 at 15:28 > To: [email protected] <[email protected]> > Subject: Re: Building a PLC4X and IoTDB Historian? > Hi all, > > I guess I would sort of also vote for a separate Project. Would however only > make sense if some people from both of our projects would join in. > > And name-wise … one of the names of Historic Historians (Ideally a Greek one) > would make it into my top 10 ;-) … sort of like Hudson or Jenkins are > “famous” Buttler names ;-) > https://en.wikipedia.org/wiki/List_of_Greek_historiographers > https://en.wikipedia.org/wiki/List_of_historians > > I think anything but a turnkey-ready solution will not be accepted by anyone > in the Automation industry. > > Chris > > > > > From: Xiangdong Huang <[email protected]> > Date: Sunday, 27. November 2022 at 14:05 > To: [email protected] <[email protected]> > Subject: Re: Building a PLC4X and IoTDB Historian? > If we just provide a solution, or demonstration, then either is ok. > If we want to provide an "one-box thing" (even without GUI), +1 for a > new project. > > > I don’t even think it would be bad if an entity like Timecho would add > > enterprise offerings, because I know, that the industry won’t “buy” > > something, if there’s no commercial support or anyone, they can throw money > > at, even if it’s free (That might even make them more skeptical). > Yes, indeed. :D > > Best, > ----------------------------------- > Xiangdong Huang > > Ben Hutcheson <[email protected]> 于2022年11月27日周日 18:38写道： > > > > Hi, > > > > I think it's a great idea. > > > > I'd vote for having a separate project for something like this, just so > > it's clear that it is a historian and people can build it separately. I > > wouldn't want it hidden away in an obscure folder in the PLC4X repo. > > > > Cake does go with Cafe. > > > > Ben > > > > > > On Sun, Nov 27, 2022 at 3:09 AM Christofer Dutz <[email protected]> > > wrote: > > > > > Hi all, > > > > > > Trying to respond to all (except the naming question ;-) ): > > > > > > @Ryan: PLC4X Doesn’t have such a queueing, but IoTDB has … so we could be > > > collecting data on one node, writing that to TSFiles using the > > > IoTDB-Client > > > (As Xaingdong mentioned). > > > These could either be online and transfer data to the server right away, > > > or operate in an offline-like mode and periodically fetch data in bursts > > > from the client (Hope that’s correct, please correct me if I’m wrong) > > > > > > @Xaiangdong: In general, PLC4X operates in an active mode, but we also > > > have a proposed and prototypically proven “passive-mode”. Here the > > > software > > > doesn’t actively participate in the data acquisition directly, but simply > > > listens to the data stream and makes data available. We also have > > > something > > > we call a “Data-Diode”, which technically makes interference impossible as > > > it lets ethernet packets flow in one direction, but nothing in the other > > > (Firewalls usually have the problem of letting packets pass in both > > > directions for established connections). We invented this, because we knew > > > we will never have validated and audited open-source software, to a level > > > that it would be certified for some of these use cases. > > > > > > So, either we can live with everything the SCADA system is already > > > requesting, or we add an active PLC4X node in the secure network, that > > > requests data, and simply ignores it, and a second – passive mode – node > > > sits outside the secure network to capture the information. > > > > > > I agree that this sort of thing needs to be a “product”. The Automation > > > Industry just doesn’t know how to work with frameworks. Ideally a one-box > > > thing. > > > Right now, most tools I have seen don’t even need the calculation or the > > > visualization. This is usually done on another level. Important would be > > > that we could be somewhat API compatible with existing products. Them > > > usually having SQL or REST APIs, should make it relatively easy to sort of > > > produce frontends to our Apache Historian, that are API compatible with > > > some existing industry products, so they are replaceable. > > > > > > If we wanted to add a visualization layer or a data-curation layer later > > > on, we should have a chat with the NiFi or StreamPipes folks as that’s > > > what > > > they already have. > > > > > > For now, I would be proposing to build something that uses PLC4X for the > > > data-acquisition, IoTDB for the storage and build a REST frontend for > > > this, > > > that’s somewhat API compatible with one of the major established products > > > and to add more on a step-by-step basis. > > > > > > I don’t even think it would be bad if an entity like Timecho would add > > > enterprise offerings, because I know, that the industry won’t “buy” > > > something, if there’s no commercial support or anyone, they can throw > > > money > > > at, even if it’s free (That might even make them more skeptical). > > > > > > > > > What do you all think? Does this make sense? If yes, where would it make > > > sense to start working on something like this? In the IoTDB project? In > > > the > > > PLC4X project, in a separate (new) project? > > > > > > > > > Chris > > > > > > > > > From: Xiangdong Huang <[email protected]> > > > Date: Sunday, 27. November 2022 at 06:02 > > > To: [email protected] <[email protected]> > > > Subject: Re: Building a PLC4X and IoTDB Historian? > > > Hi all, > > > > > > It makes sense to use PLC4X+IoTDB to build a historian software. > > > Actually, there are several users using IoTDB as their historian > > > solution... > > > > > > According to my knowledge, a historian software contains several features: > > > > > > - get data from OPC/modubs server and write to IoTDB > > > * use PLC4x + IoTDB-client. (we call this data collector) > > > > > > - (optional) there is a single-way network gateway for security. > > > * If the gateway is deployed between device and PLC4x program, then > > > we need to check whether plc4x supports that. > > > * If the gateway is deployed between IoTDB-client and IoTDB, then we > > > need to check wheter IoTDB-client supports that. > > > > > > - Calculation Engine, e.g., we want to write C to IoTDB when we > > > collect A and B from a device (for example, C=A+B/2 ) > > > * IoTDB's trigger supports that but we need a Drag-and-drop > > > programming GUI. (maybe integrating some open source projects, like > > > Red-node is a good idea) > > > > > > - get the latest data from IoTDB for visualization (usually users need > > > to draw a process flow diagram, and put the latest data on each device > > > icon on the diagram) > > > * IoTDB supports getting latest data. but we need a new software for > > > drawing the diagram (I do not know if there is any open source project > > > for this. In our real applications, we ususally buy and integrate some > > > other commerical software) > > > > > > - send alert message if the real time data meets some rules > > > * IoTDB trigger supports the rule. but currently we have no a GUI. > > > (IoTDB also supports Prometheus Alert Manager) > > > > > > - get the historical data > > > * IoTDB supports that and grafana is good at visualization > > > > > > - last but not the least, if we provide all the features to industrial > > > users, all the GUIs should be integrated into one entrance. > > > > > > > > > Best, > > > ----------------------------------- > > > Xiangdong Huang > > > School of Software, Tsinghua University > > > > > > 黄向东 > > > 清华大学 软件学院 > > > > > > Ryan Truran <[email protected]> 于2022年11月27日周日 08:18写道： > > > > > > > > Hi folks, long time lurker… > > > > > > > > A historian would be great. Does PLC4X queue data on the machine it’s > > > > running on already? > > > > > > > > A common pattern in historians is to queue data on the runtime server > > > until > > > > data can be extracted to a sql server, preventing data loss, and > > > > reducing > > > > the need for redundancy. > > > > > > > > There are a ton of edge cases to watch out for which I can elaborate on > > > > further. > > > > > > > > I’m a former Industrial Controls Engineer turned software dev and can > > > help > > > > out where needed. > > > > > > > > -Ryan > > > > > > > > On Sat, Nov 26, 2022 at 10:24 AM Otto Fowler <[email protected]> > > > > wrote: > > > > > > > > > Would you name it “Cake”? > > > > > > > > > > From: Christofer Dutz <[email protected]> > > > > > <[email protected]> > > > > > Reply: [email protected] <[email protected]> < > > > [email protected]> > > > > > Date: November 26, 2022 at 10:36:23 > > > > > To: [email protected] <[email protected]> <[email protected] > > > >, > > > > > [email protected] <[email protected]> <[email protected]> > > > > > Subject: Building a PLC4X and IoTDB Historian? > > > > > > > > > > Hi all, > > > > > > > > > > I am currently thinking, if it wouldn’t be a good idea to build > > > something > > > > > like an open-source Historian based on PLC4X and IoTDB. > > > > > > > > > > For those of you, who don’t know what a Historian is. In the > > > manufacturing > > > > > industry they have these extremely expensive servers (usually you buy > > > them > > > > > as a bundle of hard- and software). > > > > > These servers are nothing else than a really crappy and brutally > > > expensive > > > > > Database for storing Time-Series data. > > > > > > > > > > The thing is most commercial products are currently really having > > > trouble > > > > > to keep up with the increasing amount of Data being sent. > > > > > > > > > > So instead of suggesting building an Historian at each customer’s > > > site, I > > > > > thought: Perhaps a ready-to-use solution based on open-source would be > > > a > > > > > good idea. > > > > > Not 100% sure where I’d locate such an initiative, but I would tend to > > > see > > > > > it more on the Database side. > > > > > > > > > > I think IoTDB would be the perfect storage system, all we seem to need > > > is > > > > > some sort of rest-interface that matches the industry standards for > > > > > querying the information and on the other side something like PLC4X to > > > fill > > > > > the database. > > > > > > > > > > Here some examples: > > > > > > > > > > > > > https://www.ge.com/digital/documentation/historian/version80/c_historian_apis_overview.html > > > > > > > > https://cdn.logic-control.com/docs/aveva/historian/HistorianRetrieval.pdf > > > > > > > > > > What do you folks think? > > > > > > > > > > > > > > > Chris > > > > > > > >
