It is probably worth thinking about getting *new* processors as record processors, as in having a developer focus as well. If we can think about what we can do to make that better ( we started a discussion on record / service use in the archetype ) that would be worth it.
We can definitely make it easier to build record based processors, new record readers, new record writers and custom schema registry impls. On February 25, 2019 at 15:12:05, Andy LoPresto ([email protected]) wrote: I think there are legitimate use cases for the “legacy” approaches and we should not deprecate them. However, I do think there can be better education and gentle guidance of new users to prefer the record-oriented processors over the legacy processors when appropriate. Whether this is a linked note in the processor description shown in the Add Processor dialog, improvement documentation on the website, wizard/walkthroughs, etc. is certainly a good topic for conversation here. The ConvertXtoY processors should definitely be deprecated. Andy LoPresto [email protected] [email protected] PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 23, 2019, at 12:42 PM, Bryan Bende <[email protected]> wrote: > > One thing I would add is that in the 1.9.0 release there is now schema > inference built in so that you can just start using the record processors > without having a schema. > > That being said I am neutral about deprecating the non-record processors > for source and destination systems. > > The processors I would definitely be in favor of deprecating are the > conversion processors that are replaced by ConvertRecord (Avro to JSON, > JSON to Avro, csv to avro, whatever other combos) and InferAvroSchema. All > of those should be handled by ConvertRecord + the built in schema inference > option in the readers and writers. > > On Sat, Feb 23, 2019 at 1:23 PM Mike Thomsen <[email protected]> wrote: > >> Sivaprasanna, >> >> FWIW, I think there might be merit to deprecating converting to Avro, but >> the rest I think should stay. With Avro, I feel like there is intrinsic >> danger in giving people that option if they're unwilling to learn how to >> write an Avro schema. >> >> On Sat, Feb 23, 2019 at 1:21 PM Mike Thomsen <[email protected]> >> wrote: >> >>>> The number 1 thing I don't like about the Record processors is that >> they >>> require a Schema, and the complimentary processor(s?), specifically the >>> GetMongo one, does not require a schema. >>> >>> FWIW, we just added GetMongoRecord in 1.9.0, along with GridFS >> processors. >>> >>> I'll note that arguably the best reason for you to take the dive into >>> being able to use the Record API w/ Mongo is precisely that Mongo doesn't >>> even have schema on write. It's entirely possible that 9 out of 10 people >>> on your team write a date the right way you agreed upon and then the 1 >> hold >>> out does the polar opposite and you won't know until random, bizarre >>> behavior shows up. >>> >>> On Sat, Feb 23, 2019 at 12:06 PM Ryan Hendrickson < >>> [email protected]> wrote: >>> >>>> We often don't use the Record Processors because of the Schema >> requirement >>>> and complexity to use the LookupRecord processor. >>>> >>>> I'll refer to this email in the NiFi mailing list: "GetMongo - Pass-on >>>> Initial FlowFile?"... There were suggestions to use the LookupRecord >>>> processor, but ultimately it couldn't do what we needed to be done, so >> we >>>> had to string together a set of other processors. >>>> >>>> For us, it was easier to string together a set of processors than to >>>> figure >>>> out why LookupRecord, MongoDBLookupService, and InferAvroSchema wasn't >>>> getting the job done for us. >>>> /---success---> *ReplaceText* (Prepend JSON Key) >>>> ---success--> \ >>>> / >>>> \ >>>> *GetMongo* >>>> -------> *Merge Content* >>>> (Combine >>>> on Correlation Attribute Name, Binary Concat) >>>> \ >>>> / >>>> \---original---> *ReplaceText* (Prepend JSON Key) >>>> ---success--> / >>>> >>>> >>>> If they're marked as deprecated, I'd really like to see barrier to entry >>>> with the LookupRecord processors decreased. The number 1 thing I don't >>>> like about the Record processors is that they require a Schema, and the >>>> complimentary processor(s?), specifically the GetMongo one, does not >>>> require a schema. >>>> >>>> Ryan >>>> >>>> On Sat, Feb 23, 2019 at 11:39 AM Andrew Grande <[email protected]> >>>> wrote: >>>> >>>>> I'm not sure deprecating is warranted. In my experience, record based >>>>> processors are very powerful, but have a steep learning curve the way >>>> they >>>>> are in NiFi today, and, frankly, simple things should be dead simple. >>>>> >>>>> Now, moving the record UX towards an easy extreme affects this >> equation, >>>>> but e.g. I never open up a conversation with a new user by talking >> about >>>>> records, Schema Registry or NiFi Registry. >>>>> >>>>> Maybe there's something coming up which I'm not aware yet? Please >> share. >>>>> >>>>> Andrew >>>>> >>>>> On Sat, Feb 23, 2019, 7:43 AM Sivaprasanna <[email protected] >>> >>>>> wrote: >>>>> >>>>>> Team, >>>>>> >>>>>> Ever since the Record based processors were first introduced, there >>>> has >>>>>> been active development in improving the Record APIs and constant >>>>> interest >>>>>> in introducing new set of Record oriented processors. It has gone >> to a >>>>>> level where almost all the processors that deal with mainstream tech >>>>> have a >>>>>> Record based counterpart, such as the processors for MongoDB, Kafka, >>>>> RDBMS, >>>>>> HBase, etc., These record based processors have overcome the >>>> limitations >>>>> of >>>>>> the standard processors letting us build flows which are concise and >>>>>> efficient especially when we are dealing with structured data. And >>>> more >>>>>> over with the recent release of NiFi (1.9), we now have a new >> feature >>>>> that >>>>>> offers schema inference capability which even simplifies the process >>>> of >>>>>> building flows with such processors. Having said that, I'm wondering >>>> if >>>>>> this is a right time to raise the talk of deprecating processors >> which >>>>> the >>>>>> community believes has a much better record oriented counterpart, >>>>> covering >>>>>> all the functionalities currently offered by the standard processor. >>>>>> >>>>>> There are a few things that has to be talked about, like how should >>>> the >>>>>> deprecated processor be displayed in the UI, etc., but even before >>>> going >>>>>> through that route, I want to understand the community's thoughts on >>>>> this. >>>>>> >>>>>> Thanks, >>>>>> Sivaprasanna >>>>>> >>>>> >>>> >>> >> > -- > Sent from Gmail Mobile
