I think it'd be great to create one-field records for each level that doesn't exist, for things like LookupRecord and/or UpdateRecord (from the aforementioned Jira) I had envisioned the same thing. It would make it easier for other such processors to enrich/enlarge a newly-existing record level, basically an Upsert pattern (or mkdir -p :)
Regards, Matt On Tue, May 21, 2019 at 2:32 PM Mike Thomsen <[email protected]> wrote: > > All things considered, it doesn't seem like that much work. My main concern > is how to handle it if a user does something like nest the data deep into > multiple schema levels that don't exist. How should that be handled? An > exception or generating empty record declarations that have one field? > > On Tue, May 21, 2019 at 2:07 PM Matt Burgess <[email protected]> wrote: > > > Mike, > > > > Check AbstractRouteRecord, it uses the read-first-get-schema-read-rest > > pattern. However for that snippet, it is the RecordReader that is > > possibly updating the schema (currently the only thing that does this > > is schema inference), then the RecordSetWriter is created using the > > (possibly updated) schema. For your PR, you might need to update the > > schema manually and then pass that into the > > RecordSetWriterFactory.getSchema() call. > > > > Now that I think of it, you might be the first to do this > > automatic-schema-update-on-write thing (if you choose to do so). I > > thought NIFI-5524 [1] had been implemented but apparently not. > > > > Regards, > > Matt > > > > [1] https://issues.apache.org/jira/browse/NIFI-5524 > > > > > > On Tue, May 21, 2019 at 10:08 AM Mike Thomsen <[email protected]> > > wrote: > > > > > > Matt left this suggestion: > > > > > > https://github.com/apache/nifi/pull/3231#discussion_r285693972 > > > > > > What would be a good example of that pattern if I wanted to update that > > PR > > > and document the process for others? > > > > > > Thanks, > > > > > > Mike > >
