Forward slashes work in regular expression how you expect. The issue is
that your regular expression is consuming too much data. When Daffodil
evaluates the regular expression, the data Daffodil is looking at looks
like this:
A//
Your regular expression greedily matches one or more capital letters or
forward slashes. Note that delimiters are ignored when using
lengthKind="pattern". This means your pattern matches the data "A//". So
Daffodil thinks the FreeText element has the value "A//".
This is where things start going off the rails.
Because your FreeText element consumed your // characters, it means the
the terminator that the GeneralTextInfo element requires is missing.
Because it is optional (minOccurs="0"), this is not considered an error,
it just means the element does not exist, and Daffodil will backtrack.
Presumably the rest of the schema does not expect anymore data, and so
Daffodil finishes the parse successfully. But there was left over data,
so we output an error letting you know (admittedly, pretty poorly).
The fix here it to make your pattern not consume the terminator. Your
FreeText probably wants to consume A-Z and /, but stop when it hits the
double slash. This can be done with a forward lookahead, and you
probably also want the + to be non-greedy, for example:
dfdl:lengthPattern="[/A-Z]+?(?=//)"
- Steve
On 4/6/22 11:52 AM, Roger L Costello wrote:
After many hours of effort, I figured out what is causing the error. For some
reason, Daffodil does not like a forward slash in a regex:
dfdl:lengthPattern="[/A-Z]+"
The intent of that regex is to say that the input may contain a forward slash or
any uppercase letter.
That regex is contained in here:
<xs:element name="GeneralTextInfo" minOccurs="0" dfdl:initiator="GENTEXT"
dfdl:terminator="//">
<xs:complexType>
<xs:sequence dfdl:separator="/" dfdl:separatorPosition="prefix">
<xs:element name="TextIndicator" minOccurs="0" nillable="true"
type="non-zero-length-string" dfdl:lengthPattern="[A-Z ]+"/>
<xs:element name="FreeText" minOccurs="0" nillable="true"
type="non-zero-length-string" dfdl:lengthPattern="[/A-Z]+"/>
</xs:sequence>
</xs:complexType>
</xs:element>
[The actual regex for FreeText is much more complicated than what I’ve shown. It
took me a long time to distill out the part that was causing the error.]
Here is the input:
GENTEXT/FOO/A//
The error I get is:
[error] Left over data. Consumed 1504 bit(s) with at least 3040 bit(s)
remaining.
Left over data (Hex) starting at byte 189 is: (0x0d0a47454e544558...)
Left over data (UTF-8) starting at byte 189 is: (??GENTEX...)
Questions:
1. Why can’t a forward slash be used in a regex? What’s the workaround?
2. Why can’t the error message be more helpful? Why can’t Daffodil generate
this error message:
Error on line 34, column 59 of the DFDL schema. The regex in dfdl:lengthPattern
contains a forward slash, which cannot be used because [insert reason here].
Daffodil is hereby abandoning the parse of this element (FreeText) and its
parent element (GeneralTextInfo).
If I had had that error message, I could have fixed the problem in 30 seconds.
/Roger
*From:* Mike Beckerle <mbecke...@apache.org>
*Sent:* Wednesday, April 6, 2022 11:04 AM
*To:* users@daffodil.apache.org
*Subject:* [EXT] Re: Daffodil error messages are awful
I agree that every bad error message is a bug, and any error message that is
not-helpful should be reported as one.
The left-over data error you are seeing is a bit tricky. When Daffodil is
invoked to consume data from a stream, then this situation is not even an error
at all, as it is perfectly normal for a parser to parse one message from a
stream, and stop, leaving the stream positioned for the next parse call.
Only when daffodil is invoked in a context where it is clear it is intended to
consume the entire input, is this error detected at all.
What this means is that the parse ended normally, produced an infoset, but then
it was discovered that there was data left over.
To me what can be improved here is the error message text, which should say that
"parse ended normally", should indicate that an infoset was created (and display
all/part of it), and indicate that it ended without consuming all the data,
giving all positions in both bytes+optional 0..7 bits if not on a byte boundary.
On Wed, Apr 6, 2022 at 7:53 AM Roger L Costello <coste...@mitre.org
<mailto:coste...@mitre.org>> wrote:
Hi Folks,
I ran Daffodil on my DFDL schema and got this error message:
[error] Left over data. Consumed 1504 bit(s) with at least 3040 bit(s)
remaining.
Left over data (Hex) starting at byte 189 is: (0x0d0a47454e544558...)
Left over data (UTF-8) starting at byte 189 is: (??GENTEX...)
That is a really bad error message. Why did Daffodil stop consuming the
input? No idea. What is in my DFDL schema that caused the generation of the
error? No idea.
No disrespect intended, but Daffodil has the worst error messages of any
tool that I have ever encountered.
Good error messages are important. In a recent podcast Michael Kay (creator
of Saxon) talks about his emphasis on good error messages:
What makes a good product? Users must be able to understand the error
messages. People will tell you, one thing I like about Saxon is the error
messages. To me, a bad error message is something that really needs to be
fixed. Error messages are what users are dealing with every day. They are
reading my error messages. If those glare out as being unhelpful, as being
badly spelled, then that's their experience with the product, so it's
important to get it right. I put a lot of effort into those sorts of little
details. Getting good error messages it really quite an art. Do you phrase
the error message in terms of the proper terminology of the spec, or do you
use the terminology that the users are using (which might be quite wrong)?
For example, what many users call a "tag" isn't what the spec calls a tag.
They'll use "tag" to mean "element." So which word am I going to use in an
error message? It's quite hard to get that sort of thing right. Getting a
balance between a message that is technically correct and a message that
users understand, sometimes requires a fair bit of thought. And then you've
got to phrase the error message in terms of what the user was trying to do,
not what was going on internally. That again gives you a significant
challenge. So you have to think about those sorts of things.
/Roger
