Attila, I managed to take a look at this issue and just pushed up a fix for this issue. So, it will be included when we make our next release (2.5.0).
Josh Adams ________________________________ From: Adams, Joshua <jad...@tresys.com> Sent: Tuesday, December 31, 2019 12:52 PM To: Horvath, Attila J CTR (USA) <attila.j.horvath....@mail.mil> Subject: Re: [Non-DoD Source] Re: Unparsing unordered sequences behaviour Our official releases, which include binary releases for Windows and Linux, can be found here: https://daffodil.apache.org/releases/ As I mentioned earlier, we are planning on releasing version 2.5.0 in the near future. I'm not sure at this time if it will include the fix for this issue or not, so you may need to monitor progress on bug ticket. Josh Adams Apache Daffodil (incubating) | Releases<https://daffodil.apache.org/releases/> Apache Daffodil is an effort undergoing Incubation at The Apache Software Foundation (ASF), sponsored by the Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. daffodil.apache.org ________________________________ From: Horvath, Attila J CTR (USA) <attila.j.horvath....@mail.mil> Sent: Tuesday, December 31, 2019 12:19 PM To: Adams, Joshua <jad...@tresys.com> Subject: RE: [Non-DoD Source] Re: Unparsing unordered sequences behaviour Good to know - thx. Who can we contact re: a binary windows/linux distro on CD when time comes? We don't have the dev. tools work from GITHUB source. Thx attila.j.horvath....@mail.mil ahorv...@asrcfederal.com 301.225.9879 :: office/desk 301.225.9600 :: office/floor -----Original Message----- From: Adams, Joshua <jad...@tresys.com> Sent: Tuesday, December 31, 2019 12:16 PM To: Horvath, Attila J CTR (USA) <attila.j.horvath....@mail.mil> Subject: Re: [Non-DoD Source] Re: Unparsing unordered sequences behaviour All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. ________________________________ Attilla, I did some looking into the issue that you are seeing and it is not related to unordered sequences and optional elements. However, it is related to the bug that Roger Costello just reported, where separators are being suppressed when they shouldn't be if occursCountKind="expression". Please see Caution-https://issues.apache.org/jira/browse/DAFFODIL-2262 < Caution-https://issues.apache.org/jira/browse/DAFFODIL-2262 > Once that is resolved, the behaviour you are seeing should be fixed. Josh Adams ________________________________ From: Horvath, Attila J CTR (USA) <attila.j.horvath....@mail.mil> Sent: Monday, December 30, 2019 2:04 PM To: Adams, Joshua <jad...@tresys.com> Subject: RE: [Non-DoD Source] Re: Unparsing unordered sequences behaviour BTW Josh I noticed in GITHUB's bug description "... discovered an issue when the first element of an unordered sequence is optinal and does not have a maxOccurs > 1..." Don't know if it makes any difference to the proposed solution but I get the reported error not just on the FIRST element. See attached input ~.csv file, resultant parsed ~.xml file and erroneous reconstituted ~.csv file. Thx attila.j.horvath....@mail.mil ahorv...@asrcfederal.com 301.225.9879 :: office/desk 301.225.9600 :: office/floor -----Original Message----- From: Adams, Joshua <jad...@tresys.com> Sent: Monday, December 30, 2019 1:16 PM To: Horvath, Attila J CTR (USA) <attila.j.horvath....@mail.mil>; users@daffodil.apache.org Subject: Re: [Non-DoD Source] Re: Unparsing unordered sequences behaviour All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. ________________________________ Attila, If you are building from the repo (Caution-Caution-https://github.com/apache/incubator-daffodil < Caution-Caution-https://github.com/apache/incubator-daffodil > ), it should be available now. If you are looking for an official release, I believe we are planning on releasing v2.5.0, which will include this fix and many other features, in the very near future (in the next week or two I think). Josh <Caution-Caution-https://avatars3.githubusercontent.com/u/47359?s=400&v=4> < Caution-Caution-https://github.com/apache/incubator-daffodil > GitHub - apache/incubator-daffodil: Apache Daffodil (Incubating) < Caution-Caution-https://github.com/apache/incubator-daffodil > Apache Daffodil (incubating) is the open source implementation of the Data Format Description Language (DFDL), a specification created by the Open Grid Forum.DFDL is capable of describing many data formats, including textual and binary, commercial record-oriented, scientific and numeric, modern and legacy, and many industry standards. github.com ________________________________ From: Horvath, Attila J CTR (USA) <attila.j.horvath....@mail.mil> Sent: Monday, December 30, 2019 1:04 PM To: users@daffodil.apache.org <users@daffodil.apache.org> Subject: RE: [Non-DoD Source] Re: Unparsing unordered sequences behaviour When can a patch (new version) be available for public consumption? Thx v/r attila.j.horvath....@mail.mil ahorv...@asrcfederal.com 301.225.9879 :: office/desk 301.225.9600 :: office/floor -----Original Message----- From: Adams, Joshua <jad...@tresys.com> Sent: Monday, December 30, 2019 1:03 PM To: users@daffodil.apache.org Subject: [Non-DoD Source] Re: Unparsing unordered sequences behaviour All active links contained in this email were disabled. Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser. ________________________________ The pull request has now been merged. Thanks for the nice included tests Peter! Made things easy to track down and test! Josh Adams ________________________________ From: Adams, Joshua <jad...@tresys.com> Sent: Monday, December 30, 2019 10:16 AM To: Roger Costello <coste...@mitre.org>; users@daffodil.apache.org <users@daffodil.apache.org> Subject: Re: Unparsing unordered sequences behaviour I've taken a look at this and have traced the issue down to a logic bug in the Daffodil sequence code. Essentially, optional elements that are not maxOccurs > 1 were being treated as scalar elements when they should have been treated as repeating elements, as Daffodil treats all optional elements as potentially repeating. I've created a ticket and have opened a pull request with the fix. Once the request is reviewed, the fix will be pushed up to the master repo. Ticket: Caution-Caution-Caution-https://issues.apache.org/jira/browse/DAFFODIL-2261 < Caution-Caution-Caution-https://issues.apache.org/jira/browse/DAFFODIL-2261 > PR: Caution-Caution-Caution-https://github.com/apache/incubator-daffodil/pull/308 < Caution-Caution-Caution-https://github.com/apache/incubator-daffodil/pull/308 > Josh Adams <Caution-Caution-Caution-https://avatars3.githubusercontent.com/u/47359?s=400&v=4> < Caution-Caution-Caution-https://github.com/apache/incubator-daffodil/pull/308 > Fixed issue unparsing optional unordered elements by jadams-tresys * Pull Request #308 * apache/incubator-daffodil < Caution-Caution-Caution-https://github.com/apache/incubator-daffodil/pull/308 > A user discovered an issue when the first element of an unordered sequence is optinal and does not have a maxOccurs > 1. There was a logic error that was resulting in the element being treated a... github.com ________________________________ From: Adams, Joshua <jad...@tresys.com> Sent: Monday, December 30, 2019 8:19 AM To: Roger Costello <coste...@mitre.org>; users@daffodil.apache.org <users@daffodil.apache.org> Subject: Re: Unparsing unordered sequences behaviour Hello Peter and Roger, Unordered sequence support is going to be part of the official 2.50 release for Daffodil and is available if you are pulling the latest code from the Daffodil repo. That being said, the behaviour that Peter is seeing indicates that there may be a bug in the implementation. It may be related to the bug that Mike Beckerle found when unparsing choices, which is how unordered sequences are handled behind the scenes. I'm going to take a look at this today and try to figure out what is going on. Josh Adams ________________________________ From: Costello, Roger L. <coste...@mitre.org> Sent: Monday, December 30, 2019 7:01 AM To: users@daffodil.apache.org <users@daffodil.apache.org> Subject: RE: Unparsing unordered sequences behaviour Hi Peter, This web page says that Daffodil does not currently support unordered sequences: Caution-Caution-Caution-https://daffodil.apache.org/unsupported/ < Caution-Caution-Caution-https://daffodil.apache.org/unsupported/ > /Roger -----Original Message----- From: Peter Kostouros <peter.kostou...@awta.com.au> Sent: Sunday, December 29, 2019 9:24 PM To: users@daffodil.apache.org Subject: [EXT] Unparsing unordered sequences behaviour Hi I hope someone can clarify the behaviour I am seeing with respect to unparsing unordered sequences whose elements in the info set (to be unparsed) are a subset of those defined in the schema. Basically the case I have is an element that contains an unordered sequence of subelements, where each subelement is optional. Parsing such an example results in a successful transformation, however, unparsing that output results in error. A snapshot of the tmdl xsd is shown below <element name="seq2"> <complexType> <sequence dfdl:sequenceKind="unordered"> <element name="x" type="xsd:string" dfdl:initiator="X:" dfdl:terminator="%NL;" /> <element name="y" type="xsd:string" dfdl:initiator="Y:" dfdl:terminator="%NL;" /> </sequence> </complexType> </element> Unparsing something like the following results in error <seq2> <y>y0</y> </seq2> I have attached a test case (modified from daffodil-test-ibm1/src/test/resources/test-suite/tresys-contributed/) that demonstrates this, see Test Case BE004-A. In the system I am working with the error is of the form "Expected element start event for {pug}D03_Record, but received element end event for {}D_Records." for a definition like the following, where the info set being unparsed contains XML entries for D01_Record and D02_Record elements: <!-- DETAILS --> <xs:element dfdl:lengthKind="implicit" name="D_Records" minOccurs="0"> <xs:complexType> <xs:sequence dfdl:sequenceKind="unordered"> <xs:element ref="D01_Record" /> <xs:element ref="D02_Record" /> <xs:element ref="D03_Record" /> <xs:element ref="D04_Record" /> <xs:element ref="D05_Record" /> <xs:element ref="D06_Record" /> <xs:element ref="D07_Record" /> <xs:element ref="D08_Record" /> <xs:element ref="D09_Record" /> <xs:element ref="D10_Record" /> </xs:sequence> </xs:complexType> </xs:element> I am aware that unordered sequences may be a work in progress. I am working with daffodil retrieved a few days ago from github, --version displays Apache Daffodil (incubating) 2.5.0 Peter This e-mail and any attachment is intended for the party to which it is addressed and may contain confidential information or be subject to professional privilege. Its transmission in not intended to place the contents into the public domain. If you have received this e-mail in error, please notify us immediately and delete the email and all copies. AWTA Ltd does not warrant that this e-mail is virus or error free. By opening this e-mail and any attachment the user assumes all responsibility for any loss or damage resulting from such action, whether or not caused by the negligence of AWTA Ltd. The contents of this e-mail and any attachments are subject to copyright and may not be reproduced, adapted or transmitted without the prior written permission of the copyright owner.
