Please comment on this idea. The problem is that users write a schema, get "left over data" when they test it. The schema works. The schema is, as far as DFDL and Daffodil is concerned, correct. It just doesn't express what you intended it to express. It IS a correct schema, just not for your intended format.
I think Daffodil needs to save the "last failure" purely for the case where there is left-over data. Daffodil is happily ending the parse successfully but reporting it did not consume all data. In some applications where you are consuming messages from a network socket which is a byte stream, this is 100% normal behavior (and no left-over-data error would or should be issued.) In tests and anything that is "file format" oriented, left-over data is a real error. So the fact that Daffodil/DFDL says the parse ended normally without error isn't helping. In DFDL, a variable-occurrences array, the number-of-occurrences of which is determined by the data itself, always is ended if a parse fails. So long as maxOccurs has not been reached, the parse attempts another array element, and if it fails, it *suppresses that error*, backs up to the end of the prior array element (or start of array if there are no elements at all), and *discards the failure information*, then goes on to parse "the rest of the schema" meaning the stuff after the array. But what if nothing is after the array? The "suppress the error" and "discard the failure" above,.... those are a problem, because if the parse ends with left-over data, those are the "last error before the parse ended", and those *may* be relevant to why all the data was not consumed. I think we need to preserve the failure information a bit longer than we are. So with that problem in mind here's a possible mechanism to provide better diagnostics. Maybe instead of deleting it outright we put it on a queue of depth N (shallow, like 1 or 2), and as we put more failure info on that queue the failure info it pushes out the other end is discarded, but at end of processing you can look back in the parser state and see what the last N failures were, and hopefully you find there the reason for the last array ending early.? N could be set quite deep for debugging/schema-development, so you can look back through it and see the backtracking decisions in reverse chronological order as far as you need. Comments? Variants? Alternatives?
