It sounds like we want to implement something very similar to what the
Daffodil CLI calls "streaming mode". Something along the lines of this
(making guesses about what the drill API looks like based on reading this):
val input = new InputSourceDataInputStream(inputStream)
def hasNext(): Boolean = input.hasData()
def next(): DrillRecord = {
val output = new DrillRecordInfosetOutputter()
val pr = dataProcessor.parse(input, output)
output.getDrillRecord
}
So next() calls parse() once and creates a single infoset that is
projected into a Drill data structure, with each successful next/parse
call representing a single Drill record.
Depending on the schema and the input data, it is possible the entire
file could be projected into a single record. This is is likely the
common case since DFDL schemas are generally written to consume the
whole file, but that isn't always the case.
On 2023-10-12 09:48 AM, Charles Givre wrote:
Mike,
I'll add to Paul's comments. While Drill is expecting an iterator style
reader, that iterator pattern really only applies to batches. This concept
took me a while to wrap my head around, but in any of the batch reader classes,
it's important to remember that the next() method is really applying to the
batch of records and not the data source itself. In other words, it isn't a
line iterator. What that means in practice is that you can do whatever you
want in the next method until the batch is full.
I hope this helps somewhat and doesn't add to the confusion.
Best,
-- C
On Oct 12, 2023, at 12:13 AM, Paul Rogers <par0...@gmail.com> wrote:
Mike,
This is a complex question and has two answers.
First, the standard enhanced vector framework (EVF) used by most readers
assumes a "pull" model: read each record. This is where the next() comes
in: readers just implement this to read the next record. But, the code
under EVF works with a push model: the readers write to vectors, and signal
the next record. EVF translates the lower-level push model to the
higher-level, easier-to-use pull model. The best example of this is the
JSON reader which uses Jackson to parse JSON and responds to the
corresponding events.
You can thus take over the task of filling a batch of records. I'd have to
poke around the code to refresh my memory. Or, you can take a look at the
(quite complex) JSON parser, or the EVF itself to see what it does. There
are many unit tests that show this at various levels of abstraction.
Basically, you have to:
* Start a batch
* Ask if you can start the next record (which might be declined if the
batch is full)
* Write each field. For complex fields, such as records, recursively do the
start/end record work.
* Mark the record as complete.
You should be able to map event handlers to EVF actions as a result. Even
though DFDL wants to "drive", it still has to give up control once the
batch is full. EVF will then handle the (surprisingly complex) task of
finishing up the batch and returning it as the output of the Scan operator.
- Paul
On Wed, Oct 11, 2023 at 6:30 PM Mike Beckerle <mbecke...@apache.org> wrote:
Daffodil parsing generates event callbacks to an InfosetOutputter, which is
analogous to a SAX event handler.
Drill is expecting an iterator style of calling next() to advance through
the input, i.e., Drill has the control thread and expects to do pull
parsing. At least from the code I studied in the format-xml contrib.
Is there any alternative? Before I dig into creating another one of these
co-routine-style control inversions (which have proven to be problematic
for performance.
