Ping ... Just wanted to check-in on you foks here ... is there anything I can do to help finish this?
Chris -----Ursprüngliche Nachricht----- Von: Gustavo Fernandez <[email protected]> Gesendet: Mittwoch, 25. August 2021 12:10 An: dev <[email protected]> Betreff: Re: AW: AW: Nifi integration record oriented processor for reading Hi, thanks Otto --gustavo ----- Mensaje original ----- > De: "Otto Fowler" <[email protected]> > Para: "dev" <[email protected]> > Enviados: Martes, 24 de Agosto 2021 19:22:49 > Asunto: Re: AW: AW: Nifi integration record oriented processor for > reading > ok, new review up > > From: Otto Fowler <[email protected]> <[email protected]> > Reply: Otto Fowler <[email protected]> <[email protected]> > Date: August 24, 2021 at 12:30:23 > To: [email protected] <[email protected]> <[email protected]> > Subject: Re: AW: AW: Nifi integration record oriented processor for > reading > > Sorry, I’ve been crushed by IRL work, I’ll review this as soon as > possible > > From: Iñigo Angulo <[email protected]> > <[email protected]> > Reply: [email protected] <[email protected]> > <[email protected]> > Date: August 19, 2021 at 02:42:02 > To: dev <[email protected]> <[email protected]> > Cc: Gustavo Fernandez <[email protected]> <[email protected]> > Subject: Re: AW: AW: Nifi integration record oriented processor for > reading > > Hi Chris, > > I also had the confirmation from Apache that my ICLA is filed. My > username is inigoao. > > Our collegue Alvaro has also sent it, I think in few days he will > receive the confirmation. His username is asellart. > > regards, > iñigo > > ----------------------------------------- r/>Iñigo Annngulo r/> < > ZYLK.net :: consultoría.openSource r/>telf.: 74741233337 r/>Ribera de > Axpe, > 11 r/>Edificio A, modulo 201-203 r/>48950 Erandio (Bizkaia) > r/>----------------------------------------- < > > ----- Mensaje original ----- > De: "Christofer Dutz" <[email protected]> > Para: "dev" <[email protected]>, "Gustavo Fernandez" <[email protected]> > Enviados: Miércoles, 18 de Agosto 2021 14:28:00 > Asunto: AW: AW: Nifi integration record oriented processor for reading > > Hi Gustavo, > > great ... I had something in the back of my mind, but I wasn't sure > anymore ... then we're safe on that side ;-) > > Chris > > -----Ursprüngliche Nachricht----- > Von: Gustavo Fernandez <[email protected]> r/>GGGesendet: Mittwoch, 18. > August 2021 12:32 > An: [email protected] > Betreff: Re: AW: Nifi integration record oriented processor for > reading > > Hi Chris > > I sent my ICLA a month ago. The response from ASF was: > > [...] > This message acknowledges receipt of your ICLA, which has been filed > in the Apache Software Foundation records. > With this message, the PLC4X PMC has been notified that your ICLA has > been filed. > [...] > > My apache user id is zgus > > un saludo > --gustavo > > ----------------------------------------- > Gustavo Fernández > > ZYLK.net :: consultoría.openSource > Ribera de Axpe, 11 > Edificio A, modulo 201-203 > 48950 Erandio (Bizkaia) > > movil: +34 637546184 > ofic.: +34 944272119 > email: gus at zylk.net > skype: gus.zylk > > http://twitter.com/zylknet > http://www.zylk.net/web/guest/web-2-0/blog > ----------------------------------------- > > ----- Mensaje original ----- >> De: "Christofer Dutz" <[email protected]> >> Para: [email protected] >> Enviados: Miércoles, 18 de Agosto 2021 10:49:43 >> Asunto: AW: Nifi integration record oriented processor for reading > >> HI Inigo, >> r/>> great news ... I hope Otto can spare some cycleees to review the >> PR > from r/>> the NiFFFi side ... Happy to assist with merging things back. >> r/>> As it's been quite some time ... I am pretty suuure we have an >> ICLA > on r/>> file from your side, correct??? >> r/>> Chris < >> r/>> r/>> -----Ursprüngliche Nachricht----- < >> Von: Iñigo Angulo <[email protected]> >> Gesendet: Mittwoch, 18. August 2021 10:46 >> An: dev <[email protected]> >> Betreff: Re: Nifi integration record oriented processor for reading >> r/>> Hi Otto, Chris, < r/>> We have gone over the comments on the >> Nifi PR and made a commit r/>> > including revisions for all (or most) topppics. >> r/>> the last changes we did were related to reforciiing the methods >> for > r/>> datatype conversion (for the record schheema construction), and > r/>> extending the tests. About this lasst one, we changed the r/>> > org.apache.plc4x.nifi.BasePlc4xProceessoor class to protected. This > led r/>> to renaming the test claassess package to match > BasePlc4xProcessor's one r/>> > ("""org.apache.plc4x.processors.plc4x4nifi" package was renamed to > r/>>t; ""org.apache.plc4x.nifi"). This last commit caused a conflict on the > PR, but we think it can be easily corrected when doing the merge. >> r/>> At the moment, we think this nifi feature is prrretty much > completed. Of r/>> course there are things that can bbee extended yet, > which we will continue working on. >> So please, if you could take a look and check if you miss something >> r/>> > would be great. Hopefully this could be soon tested by users in the > r/>> community too and get some feedback :) < >> r/>> thank you < >> iñigo >> r/>> ----------------------------------------- < Iñigo Angulo r/>> >> ZYLK.net :: consultoría.openSource < >> telf.: 747412337 >> Ribera de Axpe, 11 >> Edificio A, modulo 201-203 >> 48950 Erandio (Bizkaia) >> ----------------------------------------- >> r/>> ----- Mensaje original ----- < >> De: "Iñigo Angulo" <[email protected]> >> Para: "dev" <[email protected]> >> Enviados: Lunes, 5 de Julio 2021 12:38:14 >> Asunto: Re: Nifi integration record oriented processor for reading >> r/>> Hi Otto, < r/>> thank you for the review of the PR and your >> cooomments. >> r/>> We have answered them, and will review the sectttions you suggested. > I r/>> would like to know if there are any tthings you want us to priorize. > r/>> Maybe you have some deadliinees for the next release or > something, and r/>> we could try too geet things done by this time. If > you miss some task we r/>> coould do here before doing the merge, please let > us know. >> r/>> iñigo < >> r/>> ----------------------------------------- < Iñigo Angulo r/>> >> ZYLK.net :: consultoría.openSource < >> telf.: 747412337 >> Ribera de Axpe, 11 >> Edificio A, modulo 201-203 >> 48950 Erandio (Bizkaia) >> ----------------------------------------- >> r/>> ----- Mensaje original ----- < >> De: "ottobackwards" <[email protected]> >> Para: "dev" <[email protected]> >> Enviados: Jueves, 24 de Junio 2021 14:40:12 >> Asunto: Re: Nifi integration record oriented processor for reading >> r/>> Great, < I’m doing the review, so you should mark it ready :) >> r/>> r/>> r/>> FFFrom: Iñigo Angulo <[email protected]> r/>> > <iangulo@@@zylk.net.invalid> >> Reply: [email protected] <[email protected]> r/>> <dev@@@ > plc4x.apache.org> >> Date: June 24, 2021 at 04:52:19 >> To: dev <[email protected]> <[email protected]> >> Subject: Re: Nifi integration record oriented processor for reading >> r/>> Hi Otto, < r/>> I have opened the new pull request, from a >> dedddicated branch in our > r/>> fork, and a commit per contributor, thhhe team members that have > been working on this. >> r/>> Please take a look, and let me know if you wanttt me to mark it >> as > ready r/>> to review. < >> r/>> iñigo < >> r/>> ----------------------------------------- br/&&>Iñigo Anngulo >> br/> < > ZYLK.net :: >> consultoría.openSource br/>telf.: 7474123337 br/>Ribera de Axpe, 11 >> r/>> > br/>Edificio A, modulo 201-203 br/&gggt;48950 Erandio (Bizkaia) >> br/>----------------------------------------- < r/>> ----- Mensaje >> original ----- < >> De: "ottobackwards" <[email protected]> >> Para: "dev" <[email protected]> >> Enviados: Jueves, 17 de Junio 2021 16:38:13 >> Asunto: Re: Nifi integration record oriented processor for reading >> r/>> That sounds great. < r/>>> On Jun 17, 2021, at 10:14, Iñigo >> Annngulo <[email protected]> > wrote: >>> br/>> Hi Otto, < >>> br/>> So we did this workaround, we re-forked the reepo and added >>> the >> changes of the nifi-record feature on a dedicated branch. I closed >> the > r/>> previous PR, will upload the new one on the follooowing days (we > are r/>> doing some last test just in case we forrggot to copy > something...) >>> br/>> I will keep you informed. < >>> br/>> regards, < >>> iñigo >>> br/>> ----------------------------------------- br/>> Iñigo Angulo >>> r/>>> > br/>> < >> br/>> ZYLK.net :: consultoría.openSource br/>> telf.: 747412337 br/>> > r/>> Ribera de Axpe, 11 br/&gggt;> Edificio A, modulo 201-203 br/>> > 48950 r/>> Erandiooo >> (Bizkaia) br/>> ----------------------------------------- < >>> br/>> ----- Mensaje original ----- < >>> De: "ottobackwards" <[email protected]> >>> Para: "dev" <[email protected]> >>> Enviados: Jueves, 10 de Junio 2021 16:14:54 >>> Asunto: Re: Nifi integration record oriented processor for reading >>> r/>>> > br/>> Yes, that sounds great. < Thannnk you br/>>> On Jun 10, 2021, at > r/>>> 03:43, I&ntiillde;igo Anggulo <[email protected]> >> wrote: >>>> br/>>> Hi Otto, < >>>> br/>>> I have tried to do the rebase of our ccommits, but I am >>>> r/>>>> > having < >> difficulties at it... br/>>> br/>>> The issue is: we forked the repo >> r/>> > on september 2020, and started mmmaking tests and commits to our fork > (on 'develop' >> branch). Now I am trying to do a rebase (using git rebase -i ID) r/>> > specifying the ID of our first commit. But when the fillle is open for > r/>> interactive mode, it gets < >> ~900 commits on the develop branch (belonging to members of the PLC4X > r/>> community). I think this happens because beforeee opening the > r/>> PullRequest, I did a 'merge upstream' with theee actual PLC4X > repo, to get the updates of the code. >> So, I understand that in the interactive mode file, I have to leave >> r/>> > all commits to 'pick' (by default), and change our cccommits of the > nifi feature to 'squash' >> except from the first one (which also remains as 'pick'). However, >> r/>> > when I tried this, many conflicts appear, almost one per each commit > (comunity members' >> commits)... >>>> I may be doing something wrong (never did a rebase before...) and I >> prefered just to ask, as I dont want to break or cause any conflict >> to > r/>> the repo code.. If you see anything Im missing plllease let me know. >>>> br/>>> As a workaround, I was thinking we coulld close the PR, >>>> re-do > r/>>>> the < >> fork of the PLC4X repo, and add the changes to the code on a >> dedicated > r/>> 'feature-nifi-record' branch. Maybe this could mmmake things > clearer... >>>> What do you think? >>>> br/>>> thank you, < >>>> iñigo >>>> br/>>> ------------------------------------------ br/>>> Iñigo >>>> r/>>>> > Angulo < >> br/>>> br/>>> ZYLK.net :: consultoría.openSource br/>>> telf.: r/>> > 747412337 br/>>&&> Ribera de Axpe, 11 br/>>> Edificio A, modulo > 201-203 rrr/>> br/>>> 48950 Erandio >> (Bizkaia) br/>>> ----------------------------------------- < >>>> br/>>> ----- Mensaje original ----- < >>>> De: "ottobackwards" <[email protected]> >>>> Para: "dev" <[email protected]> >>>> Enviados: Jueves, 27 de Mayo 2021 16:10:19 >>>> Asunto: Re: Nifi integration record oriented processor for reading > r/>>>> br/>>> Awesome. < br/>;;>> If you can, can I ask you to: < br/>>> 1. >>>> Mark the PR as ready to review in github 2. rebase or squash it to >>>> a > r/>>>> single commit and force push to youuur branch >> to clean it up >>>> br/>>> br/>>> br/>>>> On May 27, 2021, at 06:45, Iñigo Angulo >> <[email protected]> wrote: >>>>> br/>>>> Hi Otto, Chris < >>>>> br/>>>> we have finally commited the uupdates on the Nifi >>>>> Processor > r/>>>>> to < >> the Pull Request. The changes we have done are the following: >>>>> - deducing avro datatypes from PlcResponse. Here we may check the >> method (org.apache.plc4x.nifi.util.Plc4xCommon.createSchema()), in >> r/>> > order to see if it is the best way to do it. < >>>>> - we have added in the plc4j/pom.xml an "ignoredDependency" for >> org.apache.nifi:nifi-standard-nar, as it is used on runtime and was >> r/>> > rising errors during compilation. < >>>>> - we have changed onScheduled method in Plc4xSourceRecordProcessor >> (comparing to BaseProcessor), as we have included the posibility to >> r/>> > have an input connection into the processor, and indddicate the target > r/>> addressMap through flowfile attributes. TThhe addressMap is now > created in the onTrigger method. >>>>> - we have tested the performance with S7 and Modbus protocols >>>>> r/>>>>> > (using a < >> Siemens S7-1200 and Schneider M221). We will upload an updated nifi >> r/>> > template for both protocols, but regarding this, dooo you have any > r/>> testing environment to simulate PLCs??? If that is the case, we > could r/>> prepare the Processors configurratiion to match these ones > (connection Strings and addressMaps). >>>>> br/>>>> Please take a look at the code,, any suggestion will be > r/>>>>> very < >> welcome. >>>>> br/>>>> iñigo < >>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo > r/>>>>> Angulo >> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: >> r/>> > 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>> > 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>> > ------------------------------------------ < >>>>> br/>>>> ----- Mensaje original ----- < >>>>> De: "Iñigo Angulo" <[email protected]> >>>>> Para: "dev" <[email protected]> >>>>> Enviados: Miércoles, 12 de Mayo 2021 14:42:22 >>>>> Asunto: Re: Nifi integration record oriented processor for reading > r/>>>>> br/>>>> Hi Otto, Chris, < br/>>>> we have been working on the > r/>>t;;>>> prrocessor to include the logic of >> getting the values for variables from the PLC response. We tested it >> r/>> > with the < >> S7-1200 and seems to work fine, however we would like to make some >> r/>> > further tests before commiting it. < >>>>> br/>>>> Regarding the actual method whiich takes the datatype from > r/>>>>> the < >> response object, we did it in the following way: >>>>> br/>>>> //PlcReadResponse readResponse Map<String, ? extends >>>>> PlcValue> responseDataStructure = >> readResponse.getAsPlcValue().getStruct(); >>>>> for (Map.Entry<String, ? extends PlcValue> entry : >> responseDataStructure.entrySet()) { >>>>> br/>>>> if (entry.getValue() instanceoof PlcINT) { br/>>>> r/>>>>> > builder.nammme(fielldName).type().unionOf().nullBuilder().endNull().a >>>>> n d().intType().endUnion().noDefault(); >> r/>>>>> }}}else if (entry.getValue() instanceof PlcREAL) { r/>>>>> > builder.name(fieldName).tyype(().unionOf().nullBuilder().endNull().an >>>>> d ().doubleType().endUnion().noDefault(); >> r/>>>>> }}} ... and so on for the rest of the classes on the package >> (org.apache.plc4x.java.spi.values.*) >>>>> br/>>>> //the builder object is used tto build avro schema, with >> desired datatypes (for example intType()) >>>>> } >>>>> br/>>>> br/>>>> Is this the solution you had in mind?? If you >>>>> think >> there is a better way to access PlcValues, please let us know and we >> r/>> > will update it. < >>>>> br/>>>> We will upload the code soon soo that you can take a >>>>> deeper >> look. >>>>> br/>>>> thank you!! >>>>> iñigo >>>>> br/>>>> ------------------------------------------ br/>>>> Iñigo > r/>>>>> Angulo >> br/>>>> br/>>>> ZYLK.net :: consultoría.openSource br/>>>> telf.: >> r/>> > 747412337 bbbr/>>>> Ribera de Axpe, 11 br/>>>> Edificio A, modulo r/>> > 201-203 br/>>>> 48950 Erandio (Bizkaia) brr//>>>> r/>> > ------------------------------------------ < >>>>> br/>>>> ----- Mensaje original ----- < >>>>> De: "ottobackwards" <[email protected]> >>>>> Para: "dev" <[email protected]> >>>>> Enviados: Viernes, 30 de Abril 2021 14:50:11 >>>>> Asunto: Re: Nifi integration record oriented processor for reading > r/>>>>> br/>>>> Sounds liiike a good plan!! >>>>> br/>>>>> On Apr 30, 2021, at 05:26, Iñigo Angulo >> <[email protected]> wrote: >>>>>> br/>>>>> Hi Otto, Chris, < >>>>>> br/>>>>> we have been reviewingg the comments on the pull >>>>>> request, > r/>>>>>> and < >> started to think about the approach of extracting values from >> response > directly. >> During next week, we will work on this and make some updates to the >> r/>> > code with the suggestions you made. We keep you infooormed >>>>>> br/>>>>> thank you, < >>>>>> br/>>>>> iñigo < >>>>>> br/>>>>> ------------------------------------------ br/>>>>> >>>>>> Iñigo >> Angulo br/>>>>> br/>>>>> ZYLK.net :: consultoría.openSource br/>>>>> > telf.: >> 747412337 br/>>>>> Ribera de Axpe, 11 br/>>>>> Edificio A, modulo >> r/>> > 201-203 br/>>>&&>> 48950 Erandio (Bizkaia) br/>>>>> >> ----------------------------------------- < >>>>>> br/>>>>> ----- Mensaje originall ----- >>>>>> De: "Christofer Dutz" <[email protected]> >>>>>> Para: "dev" <[email protected]> >>>>>> Enviados: Viernes, 23 de Abril 2021 18:10:35 >>>>>> Asunto: AW: AW: Nifi integration record oriented processor for > r/>>>>>> reading br/>>>&gggt;> Hi Inigo, < br/>>>>> especially if you > havee a r/>>>>>> look at the KNX protocol. This < >> doesn't define the usual IEC datatypes we tried to use for all normal > r/>> PLC drivers. < >>>>>> Here we have hundreds of datatypes that don't match any other >> protocol. I think the PLCValue approach would be the simplest. >>>>>> The one thing you have to keep in mind, is that you should check >>>>>> a >> PLCValue, if it's a list (Array type) or a Structure (Which sort of >> r/>> > relates to komplex types with a more sophisticated ssstructure). >>>>>> br/>>>>> Chris < >>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- < >>>>>> Von: Iñigo Angulo <[email protected]> br/>>>>> Gesendet: >> FFreitag, 23. April 2021 15:34 >>>>>> An: dev <[email protected]> >>>>>> Betreff: Re: AW: Nifi integration record oriented processor for >> reading >>>>>> br/>>>>> Hi Otto, Chris, < >>>>>> br/>>>>> Yes, I think the approoach you propose will be best. By > r/>>>>>> now, < >> we are generating the schema ourselves. We have a record writer who >> is > r/>> in charge of reading PLC values. Schema is definnned previously > to r/>> reading the values. We build this schema gggetting the > protocol from the r/>> 'connectionString' (S7, Modbuuss) and the > specified variable type from the 'PLC resource address String' >> containing the list of variable to read. From this we deduce the r/>> > expected Avro datatype when reading, for instance, a word in S7 or a > coil in Modbus. >> br/>&ggt;>>> br/>>>>> However, as you mentioned, the oother approach >> r/>> > will be much clearerrr and useful. Ideally, getting the actual > datatype r/>> from PLCCVValue when getting the response. >> Regarding this, we tried to keep the previously described 'mapping' >> separated from the rest of the code, so that hopefully it can be r/>> > easily replaced.. < >>>>>> br/>>>>> We have done the pull rrequest, hope you can take a look > r/>>>>>> at < >> the code and let us know what you think. We will fill the ICLA >> document > too. >>>>>> br/>>>>> thank you < >>>>>> iñigo >>>>>> br/>>>>> br/>>>>> br/>>>>> >>>>>> ----------------------------------------- < Iñigo Angulo br/>>>>> > r/>>>>>&gggt; br/>>>>> ZYLK.net :: consultoría.openSource < >>>>>> telf.: 747412337 >>>>>> Ribera de Axpe, 11 >>>>>> Edificio A, modulo 201-203 >>>>>> 48950 Erandio (Bizkaia) >>>>>> ----------------------------------------- >>>>>> br/>>>>> ----- Mensaje original ----- >>>>>> De: "Christofer Dutz" <[email protected]> >>>>>> Para: "dev" <[email protected]> >>>>>> Enviados: Jueves, 22 de Abril 2021 17:12:49 >>>>>> Asunto: AW: Nifi integration record oriented processor for >>>>>> reading > r/>>>>>> br/>>>>&gggt; Hi all, < br/>>>>> Well, you get PlcValuees > from the r/>>>>>> response that wrap the < >> different datatypes. So generally you shouldn't care about the detail > type. >>>>>> br/>>>>> However, you can call ggetObject() which returns the >>>>>> core >> value the plc-value has ... so if it's the PLCValue for a Short, r/>> > getObject will return a short value. < >>>>>> br/>>>>> Does that help?? >>>>>> br/>>>>> Chris < >>>>>> br/>>>>> br/>>>>> -----Ursprüngliche Nachricht----- < >>>>>> Von: Otto Fowler <[email protected]> >>>>>> Gesendet: Donnerstag, 22. April 2021 15:21 >>>>>> An: [email protected] >>>>>> Betreff: Re: Nifi integration record oriented processor for >>>>>> r/>>>>>> > reading br/>>>>&&> So, you are generating the schema yourself, such > r/>>>&ggtt;>> that >> downstream if they inherit schema they will just get what you r/>> > generate??? And you are trying to do that by the connection string? If > r/>> so, a different way I could imagine doingg woould be to get the ’types’ > r/>> of the data from the responses themselves. This would be more generic. > r/>> The flow I could imagine ( in OnTrigger >> ): >>>>>> br/>>>>> DO READ < >>>>>> IF NOT HAS SCHEMA >>>>>> GENERATE SCHEMA FROM RESPONSE AND CACHE IN ATOMIC WRITE WITH >>>>>> r/>>>>>> > SCHEMA br/>>>>tt;> Maybe Chris can speak tto how to get the types > r/>>>&ggtt;>> from the >> responses. >>>>>> br/>>>>> br/>>>>>> On Apr 22, 2021, at 05:48, Iñigo Anguulo >> <[email protected]> wrote: >>>>>>> br/>>>>>> Hi Chris, Otto,, >>>>>>> br/>>>>>> Regarding the RRecord Processor concept, i will try to > r/>>>>>>&gggt; give >> an overview. In Nifi, information packages are called Flowfiles, and >> r/>> > these are the actual units of information that arrre exchanged between > r/>> Procesors, all along the dataflow. FFFlowfiles have two sections > where r/>> we can manage < >> data: Attributes and Content. In the "traditional" Nifi approach, you > r/>> work with both sections, extracting informatiiion from the > Content to r/>> the Attributes and viceversa to perffform operations. > This approach r/>> could have one limitation whheen you are processing > batch data (lines r/>> from a CSV file foor instance), where you need > to split each of the r/>> lines intto ddifferent Flowfiles. Thus, a > 1000 line CSV file leads to r/>&ggt; 11000 Flowfiles to process, each > of them containing a single record. >>>>>>> br/>>>>>> On later versioons of the product, they introduced the >> Record oriented approach. This approach allows you to manage multiple > r/>> records on a single FFFlowfile's Content, as long as these > records have all the same schema. >> This means that the operations defined by the Processors are applied >> r/>> > simultaneously to the whole content at once. FFFollowing with the r/>> > previous example, a 1000 line CSV file couuldd produce a single > Flowfile r/>> with a content of < >> 1000 records. br/>>>>>> br/>>>>>> To do this, Nifi uses Avro, to r/>> > serialize thee FFFlowfile's Content. Then, the Record Oriented r/>> > Processors uuse Writers and Readers to present this information in the > r/>> desiired format (such as Avro, Json, CSV, etc). Basically, with > the br/>&> record oriented approach, Nifi introduced multiple new > Processors, and r/>> also included the Record version of many of the > """old" ones. Using this r/>> Record approach, Nifi perfomance > enhances notably, specially when working with large structured information. >>>>>>> br/>>>>>> The work we didd was creating a Record Oriented >>>>>>> r/>>>>>>> > Procccessor, >> based on the previously existing one Plc4xSourceProcessor, to read >> r/>> > values from the devices. We have also included a READDDME on the r/>> > plc4x/plc4j/integrations/apache-nifi module expllaaining the Processor > r/>> configuration and giving an example. Mooreover, we put a nifi > template r/>> with a dataflow for testiing these processors, if useful. >>>>>>> br/>>>>>> Otto, regardingg the idea behind this new Processor, > r/>>>>>>>;; that >> is right. We added the writer capability to the existing >> PLC4XSourceProcessor, so that it formats the output to the desired > configuration in a record manner. >> At the actual implementation, we did this "protocol adaptation" from >> r/>> > the sintax of the particular properties on Procccessor's configuration. > r/>> FFFor example, from connection string 's7://IP:PORT', we extract > the > S7 r/>> idenifier and map vvariaable datatypes to the actual Avro > datatypes for build the record output schema. >> However, here we dont have vast experience with PLC4X libraries, and >> r/>> > for sure there will be better ways for doing this. >>>>>>> Also about the Base Processor, we were thinking that maybe the > r/>>>>>>> best < >> approach could be to have this Base Processor, and then implement >> r/>> > readers for particular protocols as Controller Serviccces. But here > r/>> also, it could be very helpful to have your opppinion. >>>>>>> br/>>>>>> Lastly, regardiing the pull request, do you have any >> documentation on br/>>>&ggt;>> how to do this? I mean, maybe you have > r/>> defined some naming br/>&gggt;>>>> conventions, or expected > structure to r/>> ffaaciliitate later work. At the br/>>>>>> present, > we have a fork of r/>> the project where we have been working on > br/&>t;>>>&ggt;> these Nifi r/>> changes. We updateed tthe content > of our fork (fetch/merge >>>>>>> upstream) about 2 weeks ago, and commited our changes to the >> 'develop' br/>>>>>> branch. Do we bettter create a new branch with >> our > commits? >> how do you br/>>>&>>> prefer to receive the code? (we are not very >> r/>> > experts on git, just in br/&gggt;>>>>> case we could cause some r/>> > problemss.....) >>>>>>> br/>>>>>> thank you in addvance >>>>>>> br/>>>>>> iñigo < >>>>>>> br/>>>>>> br/>>>>>> ----------------------------------------- < > r/>&gggt;>>>>> Iñigo Angulo br/>>>>>> ZYLK.net :: > connsultoría.openSource >>>>>>> telf.: 747412337 >>>>>>> Ribera de Axpe, 11 >>>>>>> Edificio A, modulo 201-203 >>>>>>> 48950 Erandio (Bizkaia) >>>>>>> ----------------------------------------- >>>>>>> br/>>>>>> ----- Mensaje ooriginal ----- >>>>>>> De: "Christofer Dutz" <[email protected]> >>>>>>> Para: "dev" <[email protected]> >>>>>>> Enviados: Miércoles, 21 de Abril 2021 20:01:15 >>>>>>> Asunto: AW: Nifi integration record oriented processor for >>>>>>> r/>>>>>>> > reading br/>>&gggt;>>> The more I thinnk of it, br/>>>>>> Perhaps we > r/>>>>>>> shouuld also think of potentialllyy providing some >> information on supported configuration options. >>>>>>> Wouldn't it be cool if the driver could say: "I generally have > r/>>>>>>> these < >> options and they have these datatypes and mean this" >>>>>>> Additionally, the transports could too say: "I generally have > r/>>>>>>> these < >> options and they have these datatypes and mean this" >>>>>>> br/>>>>>> I would be our StreamPipes friends would love >>>>>>> something >> like that? Right? >>>>>>> br/>>>>>> Chris < >>>>>>> br/>>>>>> br/>>>>>> -----Ursprüngliche Nachricht----- < >>>>>>> Von: Otto Fowler <[email protected]> >>>>>>> Gesendet: Mittwoch, 21. April 2021 17:46 >>>>>>> An: [email protected] >>>>>>> Betreff: Re: Nifi integration record oriented processor for > r/>>>>>>> reading br/>>&&>>>> Hi Inigo, < br/>>>>>> I’m a coommitter > on r/>>>>>>> Apache Nifi as well as PLCC44X, I would >> be happy to review your processor. >>>>>>> If I understand what you are saying correctly, you have a single >> processor which supports record writing output? >>>>>>> br/>>>>>> plc4x -> reccords >>>>>>> br/>>>>>> And that you haave, for configuration purposes for >>>>>>> that >> processor created support on a per protocol basis for configuration >> r/>> > and validation??? >>>>>>> br/>>>>>> If there is perr protocol configuration / validation > r/>>>>>>>;; etc, >> it may be better to have a base processor, and derived processors per > r/>> protocol to handle those differences. < >>>>>>> br/>>>>>> I look forward to seeing the code. >>>>>>> br/>>>>>> br/>>>>>>> On Apr 21, 2021, at 04:05, Iñigo Angulo >> <[email protected]> wrote: >>>>>>>> br/>>>>>>> Hi all,, >>>>>>>> br/>>>>>>> I am wrriting as we have been working on the Apache > r/>>>>>&&>>> Nifi >> integration part of the project. We have created a Record oriented >> r/>> > processor for reading PLC data. It is based on the prrrevious existing > r/>> SourceProcessor, but works with records, uussing a Nifi Writer > (such as r/>> Avro, Json, and so on) to writte data on flowfiles > content. br/>>>>>>> r/>&ggt; br/>>>>>>> We updated the code on our > fork with thee actual PLC4X git r/>> repo about 2 weeks ago, and > tested itt reaading values with S7 from a r/>> S7-1200 CPU from Nifi. > Alsoo, onee of our customers has recently started r/>> to use it for > validaation. br/>>>>>>> br/>>>>>>> Currently, it works r/>> with S7 > and Modbus oover TCP. This is becaause we had to write some r/>> > classes to map connectionSString annd variableList properties (sintax) > r/>> of the processoor to the actual protocol, to be able to build > then avro r/>> scchema for ooutput flowfile, taking into account > variable datatypes, br/>> etcc. We only did this for >> S7 and Modbus. I am sure that there is a better way to do this, so at > r/>> this point you maybe could take a look to find theee best > solution and r/>> avoid needing to do this mapping. br/&ggtt;>>>>>> > br/>>>>>>> If you find br/>> this useful, we could do a ppull request to the > main PLC4x repo. > Let r/>> us know what you think. br/>>>>>&ggt;>t;> br/>>>>>>> best > regards, >>>>>>>> iñigo >>>>>>>> br/>>>>>>> ------------------------------------------ >>>>>>>> Iñigo Angulo >>>>>>>> br/>>>>>>> ZYLK.neet :: consultoría.openSource >>>>>>>> telf.: 747412337 >>>>>>>> Ribera de Axpe, 11 >>>>>>>> Edificio A, modulo 201-203 >>>>>>>> 48950 Erandio (Bizkaia) > > >>>>>> -----------------------------------------
