I took a quick look at the public NITF schema yesterday, and the potential there for unparser deadlocks is large. There is simply a bunch of nested stored-length stuff going on in that format.
I think a test data XML for unparsing which has one or more of every kind of element and sub-element in that schema is needed or we won't know if we've found them all. I recently created online example DFDL schemas that illustrate the technique for avoiding these sorts of length-field-related circularities, which is this: - For any element with lengthKind explicit and an expression that uses prior fields, that element must have complex type, and a single element child with dfdl:lengthKind 'implcit'. - This gives you two elements. The outer has its length computed from an expression, and the inner has its length rolled up from what is inside it. - OutputValueCalc expressions always want to refer to this inner element, not the outer one with the length expression. The examples are: https://github.com/DFDLSchemas/envelope-payload and https://github.com/DFDLSchemas/tcpMessage On Thu, Feb 15, 2024 at 3:49 PM Lara Blatchford < lara.blatchf...@nteligen.com> wrote: > Steve - thanks for this suggestion, it did the trick. I see that the > outputValueCalc in question is commented out in the latest version of the > schema, is this issue avoided by other schema changes in that version? I > don't have easy access to get the latest version and try it, but I wanted > to make sure an issue didn't need to be entered against the latest. > > Thanks again for your help, > Lara > > -----Original Message----- > From: Steve Lawrence <slawre...@apache.org> > Sent: Wednesday, February 14, 2024 3:21 PM > To: users@daffodil.apache.org > Subject: Re: circular deadlock on NITF unparse > > This looks like a schema bug, and there are circular deadlocks. One common > way to fix this issue is to break up an outputCalc expression so that > instead of calculating the length of an element, you calculate the length > of its children and add them together. This makes for a more complicated > expression, but often times avoids circular references. In this particular > case the dfdl:outputCalc property on the > DataExtensionSegmentLengths/DataLength element can be changed from this: > > > dfdl:outputValueCalc="{ > > dfdl:valueLength(../../../DataExtensionSegment[dfdl:occursIndex()]/Data, > 'bytes') > } > > To this: > > dfdl:outputValueCalc="{ > if > > (fn:exists(../../../DataExtensionSegment[dfdl:occursIndex()]/Data/nitf:TaggedRecordExtensions)) > then > > dfdl:valueLength(../../../DataExtensionSegment[dfdl:occursIndex()]/Data/nitf:TaggedRecordExtensions, > 'bytes') > else > > dfdl:valueLength(../../../DataExtensionSegment[dfdl:occursIndex()]/Data/DESUserDefinedData, > 'bytes') > } > > So instead of evaluating the length of just the Data element, it is > changed to get the length of the TaggedRecordExtensions element or the > DESUserDefinedData element depending on which one exists (since they are in > a choice). > > You can see we do similar things to avoid deadlocks when calculating the > lengths of various HeaderLength elements in the schema. > > > On 2024-02-14 02:33 PM, Lara Blatchford wrote: > > I’m using an older version of the NITF schema from github (attached) and > > can parse the attached NITF file successfully. On unparse I get the > > circular deadlock error below. I’m using Daffodil 3.5.0. > > > > How can I determine whether this is an issue with the data itself or a > > bug in Daffodil as I see other open issues regarding circular deadlocks? > > > > Runtime Schema Definition Error: Expressions/Unparsers are circularly > > deadlocked (mutually defined): - target length for Data expr > > <EvaluatableExpression eName='DataExtensionSegment/Data' expr='{ > > ../../Header/DataExtensionSegmentLengths[dfdl:occursIndex()]/DataLength > > }' /> - target length for DESUserDefinedData expr > > <EvaluatableExpression > > eName='DataExtensionSegment/Data/DESUserDefinedData' expr='{ > > > ../../../Header/DataExtensionSegmentLengths[dfdl:occursIndex()]/DataLength > }' /> - SuspendableExpression(DataLength, expr={ > dfdl:valueLength(../../../DataExtensionSegment[dfdl:occursIndex()]/Data, > 'bytes') }) Schema context: Data Location line 781 column 16 in > nitf.dfdl.xsd Data location was preceding byte 1074 > > > > Thank you! > > > > Lara Blatchford > > > > Principal Engineer > > > > 240-341-1421 > > > >
