Mike, This is a complex question and has two answers.
First, the standard enhanced vector framework (EVF) used by most readers assumes a "pull" model: read each record. This is where the next() comes in: readers just implement this to read the next record. But, the code under EVF works with a push model: the readers write to vectors, and signal the next record. EVF translates the lower-level push model to the higher-level, easier-to-use pull model. The best example of this is the JSON reader which uses Jackson to parse JSON and responds to the corresponding events. You can thus take over the task of filling a batch of records. I'd have to poke around the code to refresh my memory. Or, you can take a look at the (quite complex) JSON parser, or the EVF itself to see what it does. There are many unit tests that show this at various levels of abstraction. Basically, you have to: * Start a batch * Ask if you can start the next record (which might be declined if the batch is full) * Write each field. For complex fields, such as records, recursively do the start/end record work. * Mark the record as complete. You should be able to map event handlers to EVF actions as a result. Even though DFDL wants to "drive", it still has to give up control once the batch is full. EVF will then handle the (surprisingly complex) task of finishing up the batch and returning it as the output of the Scan operator. - Paul On Wed, Oct 11, 2023 at 6:30 PM Mike Beckerle <mbecke...@apache.org> wrote: > Daffodil parsing generates event callbacks to an InfosetOutputter, which is > analogous to a SAX event handler. > > Drill is expecting an iterator style of calling next() to advance through > the input, i.e., Drill has the control thread and expects to do pull > parsing. At least from the code I studied in the format-xml contrib. > > Is there any alternative? Before I dig into creating another one of these > co-routine-style control inversions (which have proven to be problematic > for performance. >
