Re: [VOTE] Release Apache Daffodil SBT Plugin 1.1.0-rc1

[Adams, Joshua]

+1 (binding)

I checked:

[OK] Link in email are correct
[OK] Verified checksum and signatures are correct on source download
[OK] Verified all built in tests pass

Josh Adams

From: Kilo, Olabusayo 
Sent: Thursday, July 18, 2024 11:59 AM
To: dev@daffodil.apache.org 
Subject: Re: [VOTE] Release Apache Daffodil SBT Plugin 1.1.0-rc1

+1 (binding)

I checked:

[OK] links in email all correct
[OK] verified source downloadable
[OK] verified release notes are error free


--
Lola Kilo

From: Steve Lawrence 
Sent: Tuesday, July 16, 2024 12:07 PM
To: dev@daffodil.apache.org 
Subject: Re: [VOTE] Release Apache Daffodil SBT Plugin 1.1.0-rc1

+1 (binding)

I checked:

[OK] hashes and signatures of source are correct
[OK] signature of git tag is correct
[OK] source release matches git tag
[OK] source compiles and all tests pass
[OK] jars built from source are exactly the same as published jars (with
SOURCE_DATE_EPOCH=1721045287)
[OK] src and jars include correct LICENSE/NOTICE
[OK] RAT check passes
[OK] no unexpected binaries in source
[OK] ~80 public and private DFDL schema projects updated to use this and pass


On 2024-07-15 09:26 AM, Steve Lawrence wrote:
> Hi all,
>
> I'd like to call a vote to release Apache Daffodil SBT Plugin 1.1.0-rc1.
>
> All distribution packages, including signatures, digests, etc. can be found 
> at:
>
> https://dist.apache.org/repos/dist/dev/daffodil/daffodil-sbt/1.1.0-rc1/
>
> Staging artifacts can be found at:
>
> https://repository.apache.org/content/repositories/orgapachedaffodil-1044/
>
> This release has been signed with PGP key 36F3494B033AE661, corresponding to
> slawre...@apache.org, which is included in the KEYS file here:
>
> https://downloads.apache.org/daffodil/KEYS
>
> The release candidate has been tagged in git with v1.1.0-rc1.
>
> For reference, here is a list of all resolved issues in the 1.1.0 milestone:
>
> https://github.com/apache/daffodil-sbt/milestone/2?closed=1
>
> For a summary of the changes in this release, see:
>
> https://daffodil.apache.org/sbt/1.1.0/
>
> Please review and vote. The vote will be open for at least 72 hours (Thursday,
> 18 July 2024, 9:30am EDT).
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove (and reason why)

Re: [DISCUSS] Release Daffodil SBT Plugin 1.1.0

[Adams, Joshua]

Agreed.  It might come together quickly but there can definitely be some 
difficulties for dealing with things like hiddenGroupRef's, tracking hits of 
particular enum values, etc.

Josh

From: Steve Lawrence 
Sent: Wednesday, July 10, 2024 11:54 AM
To: dev@daffodil.apache.org 
Subject: Re: [DISCUSS] Release Daffodil SBT Plugin 1.1.0

I think some sort of coverage capabilitt would be a great addition to the
plugin. I'm not sure if we should postpone 1.1.0 until it's done--sounds like
there's still a decent amount of work to do to generate coverage reports--but
that sounds great for a future release.

On 2024-07-10 11:35 AM, Adams, Joshua wrote:
> I know there has been discussion on creating a DFDL schema coverage analysis 
> in Daffodil itself (https://issues.apache.org/jira/browse/DAFFODIL-2815), but 
> with some work I've been doing on message generation I realized I'm basically 
> half way to doing schema coverage analysis as well as I am already iterating 
> through all elements of the schema and searching infosets for a project.  I 
> was thinking some of this code could be extracted into a plugin and then add 
> some code for generating the coverage report.  Don't know if it makes sense 
> to add to the main sbt-daffodil plugin or not, but figured it might be worth 
> bringing it up.
>
> To be clear, what is discussed in DAFFODIL-2815 is most likely a more 
> thorough solution, but the solution I'm discussing could be done pretty 
> quickly without any changes to Daffodil and would also work with generic XML 
> schemas as well.
>
> Josh
> 
> From: Steve Lawrence 
> Sent: Wednesday, July 10, 2024 8:04 AM
> To: dev@daffodil.apache.org 
> Subject: [DISCUSS] Release Daffodil SBT Plugin 1.1.0
>
> In addition to defaulting to the recently released Daffodil 3.8.0, the 
> Daffodil
> SBT Plugin has a number of bug fixes, new features, and quality of life
> enhancements. Now is probably a good time for another release of the plugin.
>
> There are three open PRs that are fairly small and worth including and waiting
> on additional +1's:
>
> #38: Add daffodilBuildsCharset setting
>
> #40: Improve error detection and diagnostics
>
> #41: Support compiling main schemas found in a jar
>
> Any additional changes wanted for 1.1.0?
>
> I'll volunteer to be the release manager. If there are no other changes I'll
> plan start the release process next week once the remaining PRs are merged.
>
> - Steve
>

Re: [DISCUSS] Release Daffodil SBT Plugin 1.1.0

[Adams, Joshua]

I know there has been discussion on creating a DFDL schema coverage analysis in 
Daffodil itself (https://issues.apache.org/jira/browse/DAFFODIL-2815), but with 
some work I've been doing on message generation I realized I'm basically half 
way to doing schema coverage analysis as well as I am already iterating through 
all elements of the schema and searching infosets for a project.  I was 
thinking some of this code could be extracted into a plugin and then add some 
code for generating the coverage report.  Don't know if it makes sense to add 
to the main sbt-daffodil plugin or not, but figured it might be worth bringing 
it up.

To be clear, what is discussed in DAFFODIL-2815 is most likely a more thorough 
solution, but the solution I'm discussing could be done pretty quickly without 
any changes to Daffodil and would also work with generic XML schemas as well.

Josh

From: Steve Lawrence 
Sent: Wednesday, July 10, 2024 8:04 AM
To: dev@daffodil.apache.org 
Subject: [DISCUSS] Release Daffodil SBT Plugin 1.1.0

In addition to defaulting to the recently released Daffodil 3.8.0, the Daffodil
SBT Plugin has a number of bug fixes, new features, and quality of life
enhancements. Now is probably a good time for another release of the plugin.

There are three open PRs that are fairly small and worth including and waiting
on additional +1's:

   #38: Add daffodilBuildsCharset setting

   #40: Improve error detection and diagnostics

   #41: Support compiling main schemas found in a jar

Any additional changes wanted for 1.1.0?

I'll volunteer to be the release manager. If there are no other changes I'll
plan start the release process next week once the remaining PRs are merged.

- Steve

Annotations on group definitions

[Adams, Joshua]

Currently working on https://issues.apache.org/jira/browse/DAFFODIL-2668 and 
from the discussion there it seems that annotations should not be allowed on 
group definitions, only on group references or the various children of a group 
definition.  Just wanted to double check to make sure that this is accurate as 
we specifically have tests (sequence_group_with_annotation_01 and 
choice_group_with_annotation_01 in SequenceGroup.tdml) that verify that 
annotations are allowed on sequence group definitions and choice group 
definitions.

Just wanted to get clarification if things like


  doc
  ...


should be allowed or not.

Josh

Re: Issues with runtime2 tests

[Adams, Joshua]

Doing everything by had seems to work fine and I simply get a return code of 0 
when running the unparse command.  I'm running on a standard AMD64 chip running 
Fedora 37 Workstation.  The other machine that I've tested on is also running 
Fedora 37 Workstation with what should be a very similar software setup and it 
works fine.

I probably need to spend some time digging around to get a better idea of why 
this is failing when being run as part of the sbt tests.

Josh

From: Interrante, John A (GE Research, US) 
Sent: Tuesday, February 21, 2023 12:54 PM
To: dev@daffodil.apache.org 
Subject: RE: Issues with runtime2 tests

Josh,

Your stack trace's error message tells me that when you ran the ex_nums test 
case with TDMLImplementation.DaffodilC, the C-compiled executable exited 
abnormally with the numeric exit code 132 instead of the normal (successful) 
exit code 0.  When a program dies because of a signal, its exit code is encoded 
as 128 + signal-number, which tells me that the executable died because it 
received the SIGILL (illegal instruction) signal.  Getting a SIGILL signal 
indicates something strange must have happened 
(https://stackoverflow.com/questions/7901867/what-causes-signal-sigill).

Is there anything unusual about your machine or is it a standard Intel/AMD 
64-bit processor running standard Ubuntu 20.04 or 22.04 LTS with the system C 
compiler, libs, etc.?   You may have to hand-generate the C code from the 
ex_nums schema (daffodil generate c -s ex_nums.dfd,xsd), hand-compile the 
executable (make -C c), and see if you can reproduce the SIGILL yourself 
(c/daffodil unparse infosets/ex_nums.dat.xml -o c/ex_num.dat).  If you do 
reproduce the signal, then you can run the executable under the gdb debugger 
and try to determine how it got the signal.

John

-Original Message-----
From: Adams, Joshua 
Sent: Tuesday, February 21, 2023 8:34 AM
To: dev@daffodil.apache.org
Subject: EXT: Issues with runtime2 tests

WARNING: This email originated from outside of GE. Please validate the sender's 
email address before clicking on links or attachments as they may not be safe.

I've been sitting on this a while as I think it's most likely just a missing 
dependency on my system, but I've been getting some errors on a few of the 
runtime2 tests.  For example:

error] Test org.apache.daffodil.runtime2.TestExNums.c_ex_nums failed: 
org.apache.daffodil.tdml.TDMLExceptionImpl: (Implementation: daffodilC) 
UnparseError: Unparse Error: Result of 
/tmp/daffodilC14041106780301072313/c/daffodil...: 132 [error] , took 4.447 sec
[error] at 
org.apache.daffodil.tdml.TDMLException$.apply(TDMLException.scala:36)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.doOnePassRoundTripUnparseExpectSuccess(TDMLRunner.scala:1204)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.runParseExpectSuccess(TDMLRunner.scala:1354)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.$anonfun$runProcessor$2(TDMLRunner.scala:989)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.$anonfun$runProcessor$2$adapted(TDMLRunner.scala:979)
[error] at scala.util.Either$RightProjection.foreach(Either.scala:652)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.runProcessor(TDMLRunner.scala:979)
[error] at org.apache.daffodil.tdml.TestCase.run(TDMLRunner.scala:922)
[error] at 
org.apache.daffodil.tdml.DFDLTestSuite.runOneTest(TDMLRunner.scala:452)
[error] at 
org.apache.daffodil.tdml.Runner.runOneTest(RunnerFactory.scala:229)
[error] at 
org.apache.daffodil.tdml.Runner.runOneTest(RunnerFactory.scala:235)
[error] at 
org.apache.daffodil.runtime2.TestExNums.c_ex_nums(TestExNums.scala:48)
[error] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
[error] at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.lang.reflect.Method.invoke(Method.java:566)
[error] ...
[error] Caused by: Unparse Error: Result of 
/tmp/daffodilC14041106780301072313/c/daffodil...: 132 [error] Caused by: 
os.SubprocessException: Result of 
/tmp/daffodilC14041106780301072313/c/daffodil...: 132
[error] at os.proc.call(ProcessOps.scala:87)
[error] at 
org.apache.daffodil.processor.tdml.Runtime2TDMLDFDLProcessor.unparse(Runtime2TDMLDFDLProcessor.scala:197)
[error] at 
org.apache.daffodil.processor.tdml.Runtime2TDMLDFDLProcessor.unparse(Runtime2TDMLDFDLProcessor.scala:217)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.doOnePassRoundTripUnparseExpectSuccess(TDMLRunner.scala:1201)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.runParseExpectSuccess(TDMLRunner.scala:1354)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.$anonfun$runProcessor$2(TDMLRunner.scala:989)
[error] at 
org.apache.daffodil.tdml.ParserTestC

Issues with runtime2 tests

[Adams, Joshua]

I've been sitting on this a while as I think it's most likely just a missing 
dependency on my system, but I've been getting some errors on a few of the 
runtime2 tests.  For example:

error] Test org.apache.daffodil.runtime2.TestExNums.c_ex_nums failed: 
org.apache.daffodil.tdml.TDMLExceptionImpl: (Implementation: daffodilC) 
UnparseError: Unparse Error: Result of 
/tmp/daffodilC14041106780301072313/c/daffodil…: 132
[error] , took 4.447 sec
[error] at 
org.apache.daffodil.tdml.TDMLException$.apply(TDMLException.scala:36)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.doOnePassRoundTripUnparseExpectSuccess(TDMLRunner.scala:1204)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.runParseExpectSuccess(TDMLRunner.scala:1354)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.$anonfun$runProcessor$2(TDMLRunner.scala:989)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.$anonfun$runProcessor$2$adapted(TDMLRunner.scala:979)
[error] at scala.util.Either$RightProjection.foreach(Either.scala:652)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.runProcessor(TDMLRunner.scala:979)
[error] at org.apache.daffodil.tdml.TestCase.run(TDMLRunner.scala:922)
[error] at 
org.apache.daffodil.tdml.DFDLTestSuite.runOneTest(TDMLRunner.scala:452)
[error] at 
org.apache.daffodil.tdml.Runner.runOneTest(RunnerFactory.scala:229)
[error] at 
org.apache.daffodil.tdml.Runner.runOneTest(RunnerFactory.scala:235)
[error] at 
org.apache.daffodil.runtime2.TestExNums.c_ex_nums(TestExNums.scala:48)
[error] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
[error] at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.lang.reflect.Method.invoke(Method.java:566)
[error] ...
[error] Caused by: Unparse Error: Result of 
/tmp/daffodilC14041106780301072313/c/daffodil…: 132
[error] Caused by: os.SubprocessException: Result of 
/tmp/daffodilC14041106780301072313/c/daffodil…: 132
[error] at os.proc.call(ProcessOps.scala:87)
[error] at 
org.apache.daffodil.processor.tdml.Runtime2TDMLDFDLProcessor.unparse(Runtime2TDMLDFDLProcessor.scala:197)
[error] at 
org.apache.daffodil.processor.tdml.Runtime2TDMLDFDLProcessor.unparse(Runtime2TDMLDFDLProcessor.scala:217)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.doOnePassRoundTripUnparseExpectSuccess(TDMLRunner.scala:1201)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.runParseExpectSuccess(TDMLRunner.scala:1354)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.$anonfun$runProcessor$2(TDMLRunner.scala:989)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.$anonfun$runProcessor$2$adapted(TDMLRunner.scala:979)
[error] at scala.util.Either$RightProjection.foreach(Either.scala:652)
[error] at 
org.apache.daffodil.tdml.ParserTestCase.runProcessor(TDMLRunner.scala:979)
[error] at org.apache.daffodil.tdml.TestCase.run(TDMLRunner.scala:922)
[error] at 
org.apache.daffodil.tdml.DFDLTestSuite.runOneTest(TDMLRunner.scala:452)
[error] at 
org.apache.daffodil.tdml.Runner.runOneTest(RunnerFactory.scala:229)
[error] at 
org.apache.daffodil.tdml.Runner.runOneTest(RunnerFactory.scala:235)
[error] at 
org.apache.daffodil.runtime2.TestExNums.c_ex_nums(TestExNums.scala:48)
[error] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
[error] at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error] at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error] at java.lang.reflect.Method.invoke(Method.java:566)

Any idea's what I'm missing?  I've tested on a different system and everything 
works fine so I'm leaning towards some dependency that I'm missing that we may 
need to document.

Josh

3.5.0-SNAPSHOT DFDL Regression Test issues

[Adams, Joshua]

These are the results I'm seeing when I try to run the dfdl-regression-test 
against 3.5.0-SNAPSHOT with the current head of the 'main' branch.

[pass] (01/62) dfdl-agnosc-remedy
[pass] (02/62) dfdl-army-drrs-lh
[pass] (03/62) dfdl-asterix-raytheon
[fail] (04/62) dfdl-bmp
[pass] (05/62) dfdl-cef
[pass] (06/62) dfdl-cobol1
[pass] (07/62) dfdl-csv
[pass] (08/62) dfdl-disv6
[pass] (09/62) dfdl-edifact
[pass] (10/62) dfdl-examples-helloworld
[fail] (11/62) dfdl-examples-hexwords
[fail] (12/62) dfdl-examples-self-descriptive-data
[fail] (13/62) dfdl-examples-xslt-csv
[pass] (14/62) dfdl-geonames
[pass] (15/62) dfdl-gif
[pass] (16/62) dfdl-gmtif
[pass] (17/62) dfdl-hl7
[pass] (18/62) dfdl-ibm4690-tlog
[pass] (19/62) dfdl-icalendar
[pass] (20/62) dfdl-icd-gps-240
[pass] (21/62) dfdl-imf
[fail] (22/62) dfdl-imf-icalendar-cse
[fail] (23/62) dfdl-ipf-demo
[pass] (24/62) dfdl-ipfix
[pass] (25/62) dfdl-iso8583
[pass] (26/62) dfdl-janap-128
[pass] (27/62) dfdl-janap-128-dev
[pass] (28/62) dfdl-jpeg
[fail] (29/62) dfdl-jpeg2000
[fail] (30/62) dfdl-jreap
[pass] (31/62) dfdl-link16
[pass] (32/62) dfdl-link16-spock
[pass] (33/62) dfdl-magvar
[fail] (34/62) dfdl-mil-std-2045
[fail] (35/62) dfdl-mil-std-2045-0-0-4
[pass] (36/62) dfdl-nacha
[pass] (37/62) dfdl-nact
[fail] (38/62) dfdl-nitf
[pass] (39/62) dfdl-p8
[pass] (40/62) dfdl-pcap
[pass] (41/62) dfdl-plc4x-knxnet-ip
[fail] (42/62) dfdl-plc4x-s7
[fail] (43/62) dfdl-png
[pass] (44/62) dfdl-praat-textgrid
[pass] (45/62) dfdl-quasixml
[fail] (46/62) dfdl-raster
[pass] (47/62) dfdl-shp
[pass] (48/62) dfdl-sotf
[fail] (49/62) dfdl-stanag-5516
[fail] (50/62) dfdl-stanag-5516-with-nact
[pass] (51/62) dfdl-syslog
[pass] (52/62) dfdl-uscg-ucop-lh
[fail] (53/62) dfdl-usmtf
[fail] (54/62) dfdl-usmtf-generic
[pass] (55/62) dfdl-usmtf-mitre
[pass] (56/62) dfdl-vcard
[fail] (57/62) dfdl-vmf
[pass] (58/62) dfdl-vmf-s2s
[fail] (59/62) dfdl-vmf-spock
[pass] (60/62) gen-oilstock
[fail] (61/62) ibm-dfdl-crosstester
[pass] (62/62) udf-geout

Of the schemas that I've looked at, it seems several of them fail to compile 
due to relying on packages that either no longer exist or have been moved:

dfdl-examples-self-descriptive-data:
[error] 
/home/jadams/Programming/tresys/dfdl-regression-test/dfdl-examples/hexWords/src/test/java/TestHexWords.java:4:1:
 package org.apache.daffodil.util does not exist
[error] org.apache.daffodil.util.Misc
[error] 
/home/jadams/Programming/tresys/dfdl-regression-test/dfdl-examples/hexWords/src/test/java/TestHexWords.java:5:1:
 package org.apache.daffodil.xml does not exist
[error] org.apache.daffodil.xml.XMLUtils

dfdl-examples-xslt-csv:
[error] 
/home/jadams/Programming/tresys/dfdl-regression-test/dfdl-examples/xslt-csv/src/test/java/TestXSLTCSV.java:14:1:
 package net.sf.saxon does not exist
[error] net.sf.saxon.Transform


Others are due to various API changes, such as dfdl-imf-icalendar-cse:
[error] 
/home/jadams/Programming/tresys/dfdl-regression-test/dfdl-imf-icalendar-cse/src/test/scala/com/mitre/imf/TestFiltering.scala:68:21:
 not enough arguments for constructor XMLTextInfosetOutputter: (os: 
java.io.OutputStream, pretty: Boolean, xmlTextEscapeStyle: 
org.apache.daffodil.sapi.infoset.XMLTextEscapeStyle.Value)org.apache.daffodil.sapi.infoset.XMLTextInfosetOutputter.
[error] Unspecified value parameter pretty.
[error] val outputter = new XMLTextInfosetOutputter(pWriter)
[error] ^
[error] 
/home/jadams/Programming/tresys/dfdl-regression-test/dfdl-imf-icalendar-cse/src/test/scala/com/mitre/imf/TestFiltering.scala:71:34:
 type mismatch;
[error]  found   : java.nio.channels.ReadableByteChannel
[error]  required: org.apache.daffodil.sapi.io.InputSourceDataInputStream
[error] val pr = dataProcessor.parse(rbc, outputter)
[error]  ^
[error] 
/home/jadams/Programming/tresys/dfdl-regression-test/dfdl-imf-icalendar-cse/src/test/scala/com/mitre/imf/TestFiltering.scala:84:47:
 type mismatch;
[error]  found   : java.io.StringReader
[error]  required: java.io.InputStream
[error] val inputter = new XMLTextInfosetInputter(new StringReader(doc2));
[error]   ^
[error] 
/home/jadams/Programming/tresys/dfdl-regression-test/dfdl-imf-icalendar-cse/src/test/scala/com/mitre/imf/TestFiltering.scala:102:35:
 type mismatch;
[error]  found   : java.nio.channels.ReadableByteChannel
[error]  required: org.apache.daffodil.sapi.io.InputSourceDataInputStream
[error] val pr2 = dataProcessor.parse(rbc2, nullOutputter)

If someone could double check the regression test for me to make sure it's not 
just an issue on my system, that would be great.  Assuming it's not just an 
issue on my system, we will likely have to spend some time going through the 
regression suite and updating test code where appropriate.

Josh Adams

DAFFODIL-2615, DAFFODIL-2562, PR 931

[Adams, Joshua]

https://github.com/apache/daffodil/pull/931
https://issues.apache.org/jira/browse/DAFFODIL-2562
https://issues.apache.org/jira/browse/DAFFODIL-2615

I've pounding my head against this issue for too long now, but I think I at 
least have a somewhat firm idea of what the problem is, just not how to fix it.


Schema for reference:


  


  

  
  


  

  



  

  
  
  

  



As far as I can tell, everything gets set up correctly during the schema 
compile: PartialNextElementResolvers are created appropriately etc.  During 
runtime however, we are going through our infoset:


  0
  b
  bbb
  ccc
  1


So we go through the infoset and successfully unparse prev, key, and elt_b. We 
push elt_c onto the trdStack and it now looks like this:

trdStack: MStack(top=ex:elt_c, sequence[2], choice[2], group[2], sequence[1], 
element reference ex:dd9)

The PartialNextElementResolver successfully resolves elt_c, but since it is an 
optional element, it gets treated like an array and we end up calling 
state.inspect inside of RepeatingChildParser.shouldDoUnparser().  Inspect ends 
up advancing and asks what the nextElement is, which is "ex:post".  The issue 
is that we are just inspecting and the ex:post trd is never pushed onto the 
stack.  So, when we attempt to resolve element post, we check elt_c's resolver 
and fail, then check sequence[2] and fail, then check choice[2] where we get to 
the isRequiredStreamingUnparserEvent check in 
SeveralPossibilitiesForNextElement.maybeNextElement() and since all of 
choice[2]'s branches are Closed (correctly so I think) we determine that it is 
required.  So, when we try to find ex:post with choice[2]'s resolver we fail 
and instead of allowing things to continue, we instead return an 
UnexpecteElementErrorERD which brings everything to a halt essentially.

So, here is where I'm not sure what the correct solution is.

Should we not be calling inspect after we have already parsed the maximum 
allowed instances (1) of elt_c?

I don't think we should be pushing ex:post onto the stack at this point either, 
as later elt_c, sequence[2], choice[2], and group[2] get popped off the stack 
and post gets pushed on if we get past the UnexpectedElementErrorERD.

Is there a way to know that we are calling nextElement as a part of 
state.inspect and then not have it be required in 
SeveralPossibilitiesForNextElement?  I imagine this might cause other issues.

Let me know what you think.

Josh

Re: [EXTERNAL][VOTE] Release Apache Daffodil 3.4.0-rc2

[Adams, Joshua]

+1


  *   Verified hashes and signatures
  *   Verified that the src release builds correctly and passes all built in 
tests
  *   Verified that LICENSE, NOTICE, and README files are in all release 
artifacts and look correct
  *   Ran the release against the regression schemas and the only failures I 
saw were related to compilation failures of DFDLTestSuite, which I believe is 
expected as DFDLTestSuite became private at some point before 3.4.0.  Should 
these schemas be updated to use the newer tdml Runner class?

Josh Adams


From: Steve Lawrence 
Sent: Wednesday, November 2, 2022 6:23 PM
To: dev@daffodil.apache.org 
Subject: [EXTERNAL][VOTE] Release Apache Daffodil 3.4.0-rc2

[EXTERNAL EMAIL] DO NOT CLICK links or attachments unless you recognize the 
sender and know the content is safe.

Hi all,

I'd like to call a vote to release Apache Daffodil 3.4.0-rc2.

All distribution packages, including signatures, digests, etc. can be
found at:

https://dist.apache.org/repos/dist/dev/daffodil/3.4.0-rc2/

Staging artifacts can be found at:

https://repository.apache.org/content/repositories/orgapachedaffodil-1032/

This release has been signed with PGP key 36F3494B033AE661,
corresponding to slawre...@apache.org, which is included in the KEYS
file here:

https://downloads.apache.org/daffodil/KEYS

The release candidate has been tagged in git with v3.4.0-rc2.

For reference, here is a list of all closed JIRAs tagged with 3.4.0:

https://s.apache.org/daffodil-issues-3.4.0

For a summary of the changes in this release, see:

https://daffodil.apache.org/releases/3.4.0/

Please review and vote. The vote will be open for at least 72 hours
(Monday, 7 November 2022, 18:30 EDT).

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)

Thanks,
- Steve

Agile Delta Example EXI Program

[Adams, Joshua]

Hello all,

I've implemented a basic "Hello World" style example program using Agile 
Delta's EXI implementaion with Daffodil:

https://github.com/OpenDFDL/examples/pull/27

There is one issue in particular that I wanted to bring up.  Right now it 
cannot do schema aware EXI as it can't always resolve schemas correctly by 
default.  The Agile Delta SDK does have a SchemaResolver interface that would 
be pretty straightforward to impelment using Daffodil's DFDLCatalogResolver.  
However, the DFDLCatalogResolver is not exposed to the Java API (which is what 
this example is using).  It seems to me that the right thing to do would be to 
expose the DFDLCatalogResolver to the Java API but figured it should be brought 
up for general discussion.

DFDLCatalogResolver is implementing a Java interface under the hood 
(org.xml.sax.EntityResolver) and we expose most of the other SAX interfaces 
that Daffodil implements anyways.  DFDLCatalogResolver was also needed when 
implementing the Exificient support in the Daffodil Main program, but since 
that is written in Scala and is part of Daffodil itself, it wasn't a big deal 
to just import the DFDLCatalogResolver class.

Please let me know what you think the correct path forward is.  I'll take 
feedback either in this thread or in the pull request linked above.

Josh Adams

Re: EXI capability for Daffodil - requirements and design

[Adams, Joshua]

Here are my notes based on my work on supporting Exificient so far:

> * Daffodil APIs may need change to make use of various EXI libraries
> possible or smoother/easier.

I think this is definitely true.  Right now in my current pull request for 
adding Exificient to the CLI tool when I want to parse with Exificient using 
SAX it looks like this:

val saxXmlRdr = processor.newXMLReaderInstance
saxXmlRdr.setContentHandler(saxContentHandler)
saxXmlRdr.setProperty(XMLUtils.DAFFODIL_SAX_URN_BLOBDIRECTORY, blobDir)
saxXmlRdr.setProperty(XMLUtils.DAFFODIL_SAX_URN_BLOBSUFFIX, blobSuffix)
saxXmlRdr.parse(data)

 I feel that it could be smoothed out greatly if we could do something like:

 processor.parseWithSAX(data, saxContentHandler, saxProperties)

 Similar improvements could be made on the unparse side of things as well.

 > * Daffodil's unparser SAX API has some overhead we may want to bypass. The
> unparser is naturally a pull/StAX style of API. If EXI libraries can
> accommodate this then that may be substantially better in performance. EXI
> is all about performance after all.

I did some testing with the current SAX based approach for EXI in my pull 
request and there doesn't seem to be any difference in performance for parsing 
between EXI and regular XML infosets, but for unparsing EXI (using SAX) is 
about 3 times slower than normal XML.

Exificient has a StAX API as well so this is probably worth investigating.  
Does daffodil already support StAX or would we need to implement some sort of 
XMLStreamReader/XMLStreamWriter?

> 1) support for multiple open and closed source EXI implementations that
> are not incorporated into Daffodil as dependencies
> I know we have users who want to see tests with at least Agile Delta EXI
> (closed source) and EXIfficient.

I'm looking into Agile Delta and have requested an evaluation copy through 
their website, but I'm not sure that will give us access to their SDK.  Should 
allow us to at least verify that we can unparse an EXI file encoded by Agile 
Delta with our current implementation using Exificient though.

> 2) support for schema-unaware EXI encoding

This is how it is currently implemented in my pull request

> 3) support for schema-ware EXI encoding. This may introduce new
> requirements - e.g., unlike XML text or schema unaware, one may (I have a
> lack of EXI knowledge/experience here) need the schema (or some
> EXI-compiled flavor thereof) in order to consume such EXI. (Bunch of TBD
> here.)

This hopefully wouldn't be too difficult to implement in the CLI, the only 
thing I'm not sure about is how well the EXI libraries would handle resolving 
our schemas spread out across several files.  Should be a solvable problem 
though.  I'm thinking it would only be supported when the --schema option is 
present on the CLI (i.e. use schema-unaware if using saved parsers).

> 4)  ? TDML runner support (? is there any requirement here ? Unclear)

Only thing that might be nice to have here is a way to compare EXI infosets, 
but I'm not sure if this is really necessary.  There isn't much value in 
inspecting an EXI infoset, so long as you can verify that it unparses 
correctly, matching the original file.

> 5) CLI support to output schema-unaware exi.
>
> 5.5) CLI support to output schema-aware exi. (TBD: is this needed for CLI?
> Applications can do this from API, do we really also need to offer it from
> the CLI?)

Touched on these earlier.  Should be doable, but might be limited to 
schema-unaware for saved-parsers

> 6) Enable EXI LZW Compression feature (or not) - EXI is all about
> performance by improving the data density hence the handling overhead. We
> should do experiments measuring the on/off of options such as compression
> (a LZW-style compression feature built into EXI encoders/decoders) which is
> optionally enabled. If this improves compression with low overhead we would
> just turn it on. If the benefits are small we would not bother with it,
> but...  if it reduces size substantially, but has real measurable cost,
> then we probably need a switch for on/off. An interesting point would be
> the use of LZW compression with non-schema aware EXI vs. schema-aware EXI
> (with or without compression).

This would simply be a matter of setting the appropriate flags to the 
EXIFactory before creating the EXI SaxContentHandler.  Could easily be added to 
my existing pull request.

> 7) Unparser - API Pull support - Speculation here - do we need to create a
> standard StAX API for Daffodil unparsing so that EXI software supporting
> StAX (or any other kind of StAX software) can be used with Daffodil more
> easily.

Touched on this earlier that Exificient does have a StAX API and is worht 
investigating due to the performance overhead of SAX when unparsing.

* Examples should show both file-of-data mode, and streaming mode (many
messages on unbounded input)

* CLI exi feature must support both single-file parse/unparse, and
streaming

Re: CLI -I sax feature - remove?

[Adams, Joshua]

Funnily enough I was just having a conversation with Dave Thompson about this 
very thing.  When I mentioned that we will have a "-I exi" option for the 
command line tool, which from Daffodil's perspective will be just like using 
SAX, he asked if he will need to specify the "-I sax" option, or if that option 
will just go away.

I am in full agreement of removing the "-I sax" option as, like you said, SAX 
usage is inherently an API and I can't think of an easy way to pass a SAX 
ContentHandler over the command line.  Adding an EXI option to the Daffodil CLI 
tool can kill 2 birds with one stone in that we will be adding support for EXI 
as well as demonstrating how to pass in your own SAX ContentHandler.

Josh

From: Mike Beckerle 
Sent: Monday, July 11, 2022 3:19 PM
To: dev@daffodil.apache.org 
Subject: CLI -I sax feature - remove?

In the CLI, there is the -I option to specify infoset-type.

One of the choices is 'sax'.

This is a mistake I think. This is really "XML text by way of calling the
SAX API". It's effectively a testing mode for us.

SAX usage is inherently an API.

I believe we should remove this feature from the CLI, because it creates a
lot of confusion. It requires test-mode things to be in the main-libraries
where the CLI can find them.

If we require SAX to be used as it is intended, from applications calling
Daffodil via APIs, then all this "xml text to/from SAX-event" code all ends
up in src/test where it belongs.

Thoughts?

Re: [VOTE] Release Apache Daffodil 3.3.0-rc1

[Adams, Joshua]

+1

I have verified signatures of all packages and gave been using the 
3.3.0-SNAPSHOT extensively.

Josh Adams

On Mar 18, 2022 12:07 PM, "Interrante, John A (GE Research, US)" 
 wrote:
Hi PMC members,

I'd like to call a vote to release Apache Daffodil 3.3.0-rc1.

All distribution packages, including signatures, digests, etc. can be
found at:

https://dist.apache.org/repos/dist/dev/daffodil/3.3.0-rc1/

Staging artifacts can be found at:

https://repository.apache.org/content/repositories/orgapachedaffodil-1029/

This release has been signed with PGP key 04A735FC1A36AE84, corresponding
to jinterra...@apache.org, which is included in the KEYS file here:

https://downloads.apache.org/daffodil/KEYS

The release candidate has been tagged in git with v3.3.0-rc1.

For reference, here is a list of all closed JIRAs tagged with 3.3.0:

https://s.apache.org/daffodil-issues-3.3.0

For a summary of the changes in this release, see:

https://daffodil.apache.org/releases/3.3.0/

Please review and vote. The vote will be open for at least 72 hours
(Monday, March 21 2022, 12 Noon EST).

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)

Thanks,
John

Re: sbt exportJars causes "lengthLimitInBits.>=(0)" error when running "sbt test"

[Adams, Joshua]

I can confirm that at least sbt clean does not resolve the issue.  Didn't try a 
got clean, although I had just cloned the repo in question so I don't think it 
had gotten dirty in anyway either.

Josh

On Mar 18, 2022 1:22 PM, Steve Lawrence  wrote:
That's surprising to me. All that exportJars is supposed to to do is
include or not include certains files on the class path and use packages
jars instead.

And that assertion that is failing is beacuase the length of test data
is negative, which should never happen and I don't know how exportJars
would affect that.

It almost sounds like something got corrupted or needed to be rebuilt,
and changing exportJars allowed that to happen? Maybe there's some
conflict between using IntelliJ and sbt at the same time?

I wonder if doing an sbt clean or git clean would resolve the issue.


On 3/18/22 1:10 PM, Olabusayo Kilo wrote:
> We noticed that only when running "sbt test" in schema projects with the
> exportJar setting, the error below was generated resulting in failure of
> all the tests. Note that running the same tests via IntelliJ/Junit
> didn't reproduce this error, and the error was isolated to using sbt to
> run the tests.
>
> The cause was isolated to having the "exportJars := true" setting in
> build.sbt. Removing the setting removed the error, and got all tests
> passing. Sending this out as an FYI in case someone else runs into this
> in the future.
>
> [info] Test a.b.c started
> [error] Test a.b.c failed: org.apache.daffodil.exceptions.Abort: Usage
> error: lengthLimitInBits.>=(0)
> [error] org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:137)
> [error]
> org.apache.daffodil.tdml.processor.DaffodilTDMLDFDLProcessor.doParseWithBothApis(DaffodilTDMLDFDLProcessor.scala:290)
>
> [error]
> org.apache.daffodil.tdml.processor.DaffodilTDMLDFDLProcessor.parse(DaffodilTDMLDFDLProcessor.scala:248)
>
> [error]
> org.apache.daffodil.tdml.processor.DaffodilTDMLDFDLProcessor.parse(DaffodilTDMLDFDLProcessor.scala:253),
> took 0.003 sec
> [error] at
> org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:137)
> [error] at
> org.apache.daffodil.tdml.processor.DaffodilTDMLDFDLProcessor.doParseWithBothApis(DaffodilTDMLDFDLProcessor.scala:290)
>
> [error] at
> org.apache.daffodil.tdml.processor.DaffodilTDMLDFDLProcessor.parse(DaffodilTDMLDFDLProcessor.scala:248)
>
> [error] at
> org.apache.daffodil.tdml.processor.DaffodilTDMLDFDLProcessor.parse(DaffodilTDMLDFDLProcessor.scala:253)
>
> [error] at
> org.apache.daffodil.tdml.ParserTestCase.doParseExpectSuccess(TDMLRunner.scala:1026)
>
> [error] at
> org.apache.daffodil.tdml.ParserTestCase.runParseExpectSuccess(TDMLRunner.scala:1169)
>
> [error] at
> org.apache.daffodil.tdml.ParserTestCase.$anonfun$runProcessor$2(TDMLRunner.scala:920)
>
> [error] at
> org.apache.daffodil.tdml.ParserTestCase.$anonfun$runProcessor$2$adapted(TDMLRunner.scala:917)
>
> [error] at scala.util.Either$RightProjection.foreach(Either.scala:652)
> [error] at
> org.apache.daffodil.tdml.ParserTestCase.runProcessor(TDMLRunner.scala:917)
> [error] at org.apache.daffodil.tdml.TestCase.run(TDMLRunner.scala:870)
> [error] at
> org.apache.daffodil.tdml.DFDLTestSuite.runOneTest(TDMLRunner.scala:416)
> [error] at
> org.apache.daffodil.tdml.Runner.runOneTest(RunnerFactory.scala:191)
> [error] at
> org.apache.daffodil.tdml.Runner.runOneTest(RunnerFactory.scala:197)
>

Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance

[Adams, Joshua]

I've been looking at this some more today and I'm not seeing an easy way to get 
the enclosing sequence/group's bitOrder and apply it to the NewVariableInstance 
runtime data.  NVI has access to the AnnotatedSchemaComponent, which does have 
access to enclosing elements, but I don't think there is any access to the 
runtime data for the enclosing element.  I suppose it would be possible to read 
the raw attributes from enclosing element but I'm not sure that would work well 
if bitOrder is defined as an expression.

Josh

From: Mike Beckerle 
Sent: Tuesday, November 23, 2021 3:27 PM
To: dev@daffodil.apache.org 
Subject: Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance

Presumably the same issue would arise if dfdlx:lookahead was called from a
setVariable annotation on a sequence also.

I happen to know there is a requirement for this to reach into data at
boundaries that are NOT byte boundaries, so the dfdl:bitOrder really does
matter for this function.

I claim the right thing is for the bitOrder to come from the model group on
which the newVariableInstance or setVariable is residing.

My reasoning is that if you call dfdlx:lookAhead from the choiceDispatchKey
expression of a choice, the bitOrder is that of the choice model group.
That's how it works today.
There should be no difference between expressing a dfdl:choiceDispatchKey
that directly uses a dfdlx:lookAhead call, and one that does so by way of a
new variable instance.


  
 
  
  
 
  
 choice branches here


So I think the right answer is to just require the enclosing model group's
model-group runtime data object be passed down to the newVariableInstance
or setVariable primitive parser/unparser.

On Tue, Nov 23, 2021 at 1:05 PM Adams, Joshua 
wrote:

> I've been taking a look at this bug and am trying to figure out what the
> correct behavior for this situation would be.  Currently, attempting to
> call dfdlx:lookAhead as part of an expression for the default value of a
> newVariableInstance will result in a UsageError being thrown when
> attempting to get the bitOrder, as the VariableRuntimeData is descended
> from NonTermRuntimeData.
>
> It doesn't seem unreasonable to want to use lookAhead from inside a
> newVariableInstance, but I'm not sure there is an easy way to get the
> parent sequence/element that contains the annotation where the
> newVariableInstance is defined.  I noticed that for elements when the
> expression for byteOrder isn't defined, it just defaults to BigEndian with
> comments stating that this likely means that the element is likely
> hexBinary, in which case the byte order should be ignored.  I don't know if
> it makes sense to do something like this and just defauilt bitOrder for
> VariableRuntimeData's to MSBF, as from the examples of lookAhead that I've
> seen involve looking into a hexBinary element.
>
> Thoughts?
>
> Josh
>

Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance

[Adams, Joshua]

Out of curiosity, would look ahead work at all if it was used as the default 
value of a regular defineVariable statement or is that not allowed?  The spec 
says that the defaultValue of a defineVariable statement cannot reference the 
infoset, but using a lookAhead is a little different than referencing a forward 
path.

Josh

From: Mike Beckerle 
Sent: Tuesday, November 23, 2021 3:27 PM
To: dev@daffodil.apache.org 
Subject: Re: DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance

Presumably the same issue would arise if dfdlx:lookahead was called from a
setVariable annotation on a sequence also.

I happen to know there is a requirement for this to reach into data at
boundaries that are NOT byte boundaries, so the dfdl:bitOrder really does
matter for this function.

I claim the right thing is for the bitOrder to come from the model group on
which the newVariableInstance or setVariable is residing.

My reasoning is that if you call dfdlx:lookAhead from the choiceDispatchKey
expression of a choice, the bitOrder is that of the choice model group.
That's how it works today.
There should be no difference between expressing a dfdl:choiceDispatchKey
that directly uses a dfdlx:lookAhead call, and one that does so by way of a
new variable instance.


  
 
  
  
 
  
 choice branches here


So I think the right answer is to just require the enclosing model group's
model-group runtime data object be passed down to the newVariableInstance
or setVariable primitive parser/unparser.

On Tue, Nov 23, 2021 at 1:05 PM Adams, Joshua 
wrote:

> I've been taking a look at this bug and am trying to figure out what the
> correct behavior for this situation would be.  Currently, attempting to
> call dfdlx:lookAhead as part of an expression for the default value of a
> newVariableInstance will result in a UsageError being thrown when
> attempting to get the bitOrder, as the VariableRuntimeData is descended
> from NonTermRuntimeData.
>
> It doesn't seem unreasonable to want to use lookAhead from inside a
> newVariableInstance, but I'm not sure there is an easy way to get the
> parent sequence/element that contains the annotation where the
> newVariableInstance is defined.  I noticed that for elements when the
> expression for byteOrder isn't defined, it just defaults to BigEndian with
> comments stating that this likely means that the element is likely
> hexBinary, in which case the byte order should be ignored.  I don't know if
> it makes sense to do something like this and just defauilt bitOrder for
> VariableRuntimeData's to MSBF, as from the examples of lookAhead that I've
> seen involve looking into a hexBinary element.
>
> Thoughts?
>
> Josh
>

DAFFODIL-2587 - dfdlx:lookAhead from newVariableInstance

[Adams, Joshua]

I've been taking a look at this bug and am trying to figure out what the 
correct behavior for this situation would be.  Currently, attempting to call 
dfdlx:lookAhead as part of an expression for the default value of a 
newVariableInstance will result in a UsageError being thrown when attempting to 
get the bitOrder, as the VariableRuntimeData is descended from 
NonTermRuntimeData.

It doesn't seem unreasonable to want to use lookAhead from inside a 
newVariableInstance, but I'm not sure there is an easy way to get the parent 
sequence/element that contains the annotation where the newVariableInstance is 
defined.  I noticed that for elements when the expression for byteOrder isn't 
defined, it just defaults to BigEndian with comments stating that this likely 
means that the element is likely hexBinary, in which case the byte order should 
be ignored.  I don't know if it makes sense to do something like this and just 
defauilt bitOrder for VariableRuntimeData's to MSBF, as from the examples of 
lookAhead that I've seen involve looking into a hexBinary element.

Thoughts?

Josh

Re: [DISCUSS] Are we ready to release 3.2.0 Daffodil ?

[Adams, Joshua]

I took a look at the EDIFACT and NACHA schemas, which ar both failing because a 
predefined variable (encoding) is attempting to reset beyond it's original 
definition.  I attempted to make a simple test case to reproduce the issue by 
having a choice branch that will backtrack after reading the varible, but was 
unable to reproduce this double reset that seems to be occurring for these 
schemas.

I've got a bit of a hacky pull request up that works around this issue, but 
there may still be a deeper problem here that is causing this double reset to 
occur.

https://github.com/apache/daffodil/pull/674

Josh
[https://opengraph.githubassets.com/d874d568be0e42159e8e2f2c6d553ed0af1c753e327ba64b30110c5a73cc3f09/apache/daffodil/pull/674]
Work around issue resetting predefined vars by jadams-tresys · Pull Request 
#674 · apache/daffodil
This is a quick fix for this bug where a predefined variable is being reset 
beyond its original definition. This was occurring in the EDIFACT and NACHA 
schemas. Definitely feels a little hacky and...
github.com


From: Mike Beckerle 
Sent: Wednesday, November 3, 2021 3:51 PM
To: dev@daffodil.apache.org 
Subject: Re: [DISCUSS] Are we ready to release 3.2.0 Daffodil ?

I guess I have found issues indicating we are not quite ready for release.

These may be caused by incompatibilities of the tests, but Owl has a
regression test rig where we have every DFDL schema we can get our hands
on.

I ran this for daffodil 3.1.0 and all tests pass.

For daffodil 3.2.0-SNAPSHOT I get failures for these:

[error] (dfdl-vmf-spock / Test / test) sbt.TestsFailedException: Tests
unsuccessful
[error] (dfdl-edifact / Test / test) sbt.TestsFailedException: Tests
unsuccessful
[error] (udf-geoutil / Test / test) sbt.TestsFailedException: Tests
unsuccessful
[error] (dfdl-nacha / Test / test) sbt.TestsFailedException: Tests
unsuccessful
[error] (dfdl-link16 / Test / test) sbt.TestsFailedException: Tests
unsuccessful
[error] (dfdl-examples-hexwords / Test / test) sbt.TestsFailedException:
Tests unsuccessful

I will investigate why this is. From looking at the diagnostics, one of
them at least is due to xmlns="" changes in expected output vs. actual,
and since there was a change there in 3.2.0 I expect that may be a matter
of adjusting the test case data.

However, there is at least one bug here. As one of the failures is due to
an Assert.impossible() call executing.


On Tue, Nov 2, 2021 at 1:39 PM John Wass  wrote:

> Nothing here for 3.2.0.
>
> On Mon, Nov 1, 2021 at 3:42 PM Interrante, John A (GE Research, US) <
> john.interra...@ge.com> wrote:
>
> > I checked the Daffodil JIRA to see how many blocker, critical, and major
> > issues we have so far:
> >
> > Blocker issues: 0
> > Critical issues: 1
> > https://issues.apache.org/jira/browse/DAFFODIL-2400 - New SAX
> API
> > causes performance degradations
> > Major issues: 132
> > https://issues.apache.org/jira/browse/DAFFODIL-2574 - Cast error
> > when multiplying two unsignedBytes
> > and so on - too many to list individually
> >
> > The critical issue isn't a blocker and in fact hasn't been updated since
> > October 2020 (1 year ago).  I see Daffodil's release plan in
> >
> https://cwiki.apache.org/confluence/display/DAFFODIL/Roadmap+for+Upcoming+Releases
> > already suggests reducing the number of JIRA issues between 3.2.0 and
> 3.3.0
> > which sounds like a good plan.  Everything planned for 3.2.0 seems to be
> > already checked off.
> >
> > I looked over the instructions in
> > https://cwiki.apache.org/confluence/display/DAFFODIL/Release+Workflow
> > carefully and preparing a release candidate looks doable but still a
> > time-consuming bit of work.  Since we already have a volunteer, I'd
> rather
> > wait for another release (thanks Mike).
> >
> > The first step in the Release workflow says to create a [DISCUSS] thread
> > and allow time for discussion before creating the release candidate.
> I've
> > renamed this email's subject to start that formal thread and I'm fine
> with
> > waiting less than 72 hours if we get responses from everyone we know who
> > might be working on anything that they want to release in 3.2.0.  I plan
> to
> > make some more changes and improvements to the Daffodil C code generator
> > myself but the changes can go into 3.3.0 since I already build Daffodil
> > from source anyway.  Is Steve Lawrence or John Wass working on something
> > that should make it into 3.2.0?  Anyone else?
> >
> > John
> >
> > -Original Message-
> > From: Mike Beckerle 
> > Sent: Monday, November 1, 2021 1:44 PM
> > To: dev@daffodil.apache.org
> > Subject: EXT: Are we ready to release 3.2.0 Daffodil ?
> >
> > I believe 3.2.0 is functionally complete.
> >
> > Should we prepare a release candidate that we can vote on?
> >
> > We need someone to volunteer to be release manager.
> >
> > I am w

Re: Daffodil 3.2.0 - call for final bugs/issues?

[Adams, Joshua]

https://issues.apache.org/jira/browse/DAFFODIL-2568

I've run into this with schemas that are in use in the real world.  It only 
effects testing infrastructure in that the SAX outputter will output redundant 
namespaces that cause failures when comparing to expected results.  While the 
SAX output with redundant namespaces is functionally equivalent, this can cause 
a fair amount of headache for someone developing a schema to try and figure out 
what is going wrong when tests that look OK end up failing.

Josh

From: Mike Beckerle 
Sent: Monday, October 18, 2021 1:34 PM
To: dev@daffodil.apache.org 
Subject: Daffodil 3.2.0 - call for final bugs/issues?

Release 3.2.0 is, I think, at a state of functional completion.

Before we start discussing a release candidate, I wanted to have an email
thread about

* any near-term additional things people are trying to get in
* any serious bugs that they believe are high-priority or should be
blockers.

** I think making sure 3.2.0 works with the forthcoming vscode debugger is
certainly one task.

Re: [VOTE] Release Apache Daffodil 3.1.0-rc2

[Adams, Joshua]

+1

I checked

[OK] RPM installs and runs as expected on CentOS 8
[OK] Signatures and hashes for RPM and source are correct
[OK] Correct LICENSE/NOTICE/DISCLAIMER in src, RPM, JAR files
[OK] Source release matches git tag (minus KEYS file)
[OK] No unexpected binaries in source
[OK] Source compiles and all tests pass
[OK] RAT check passes

Josh

From: Steve Lawrence 
Sent: Monday, May 17, 2021 10:21 AM
To: dev@daffodil.apache.org 
Subject: Re: [VOTE] Release Apache Daffodil 3.1.0-rc2

+1

I checked:

[OK] hashes and signatures of source and helper binaries are correct
[OK] signature of git tag is correct
[OK] source release matches git tag (minus KEYS file)
[OK] source compiles and all tests pass (both en_US and de_DE LANG)
[OK] jars in helper binaries and the repository are exactly the same
[OK] jars built from source are exactly the same as helper binary jars
[OK] src, binaries, and jars include correct LICENSE/NOTICE
[OK] RAT check passes
[OK] no unexpected binaries in source
[OK] distributed dependencies in helper binaries are same as from maven
[OK] rpm and msi install and run with basic usage
[OK] ~60 public and private DFDL schema projects pass tests
[OK] No issues found in JavaDoc and ScalaDoc


On 5/17/21 9:55 AM, Interrante, John A (GE Research, US) wrote:
> +1
>
> I checked (for most items, using WSL2/Ubuntu 20.04 on my laptop):
>
> [OK] rpm installs correctly on Fedora Workstation 34
>
> [OK] some daffodil CLI commands work (generate c, test on runtime2 examples)
>
> [OK] signature of git tag is correct
>
> [OK] hash and signature of each download is correct
>
> [OK] src, bins, and jars include correct LICENSE/NOTICE/DISCLAIMER
>
> [OK] source release matches git tag (minus KEYS file)
>
> [OK] no unexpected binaries in source
>
> [OK] source compiles and all tests pass
>
> [OK] RAT check passes
>
> [OK] jars built from source have the same content as helper binary jars
>
> John
>
> -Original Message-
> From: Steve Lawrence 
> Sent: Friday, May 14, 2021 3:27 PM
> To: dev@daffodil.apache.org
> Subject: EXT: [VOTE] Release Apache Daffodil 3.1.0-rc2
>
> Hi all,
>
> I'd like to call a vote to release Apache Daffodil 3.1.0-rc2.
>
> All distribution packages, including signatures, digests, etc. can be found 
> at:
>
> https://dist.apache.org/repos/dist/dev/daffodil/3.1.0-rc2/
>
> Staging artifacts can be found at:
>
> https://repository.apache.org/content/repositories/orgapachedaffodil-1023/
>
> This release has been signed with PGP key 36F3494B033AE661, corresponding to 
> slawre...@apache.org, which is included in the KEYS file here:
>
> https://downloads.apache.org/daffodil/KEYS
>
> The release candidate has been tagged in git with v3.1.0-rc2.
>
> For reference, here is a list of all closed JIRAs tagged with 3.1.0:
>
> https://s.apache.org/daffodil-issues-3.1.0
>
> For a summary of the changes in this release, see:
>
> https://daffodil.apache.org/releases/3.1.0/
>
> Please review and vote. The vote will be open for at least 72 hours (Monday, 
> 17 May 2021, 4pm EST).
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove (and reason why)
>

Re: [VOTE] Release Apache Daffodil 3.1.0-rc1

[Adams, Joshua]

That seems reasonable to me.  Official releases should clearly demonstrate 
cross compatibility with IBM's implementation.

Josh

From: Steve Lawrence 
Sent: Friday, May 14, 2021 11:58 AM
To: dev@daffodil.apache.org 
Subject: Re: [VOTE] Release Apache Daffodil 3.1.0-rc1

I've found a bug in the TDML runner that causes the IBM DFDL Crosstester
to fail:

https://issues.apache.org/jira/browse/DAFFODIL-2517

This only affects the IBM DFDL Crosstester and not Daffodil, but there
isn't really a great workaround, since there is no way for IBM to report
correct final parse/unparse bit position.

I'll have a pull ready request shortly.

Thoughts on cancelling rc1 and include this fix in an rc2? The bug only
affects the IBM DFDL cross tester, but there really is no good workaround.

On 5/12/21 1:47 PM, Steve Lawrence wrote:
> Hi all,
>
> I'd like to call a vote to release Apache Daffodil 3.1.0-rc1.
>
> All distribution packages, including signatures, digests, etc. can be
> found at:
>
> https://dist.apache.org/repos/dist/dev/daffodil/3.1.0-rc1/
>
> Staging artifacts can be found at:
>
> https://repository.apache.org/content/repositories/orgapachedaffodil-1020/
>
> This release has been signed with PGP key 36F3494B033AE661,
> corresponding to slawre...@apache.org, which is included in the KEYS
> file here:
>
> https://downloads.apache.org/daffodil/KEYS
>
> The release candidate has been tagged in git with v3.1.0-rc1.
>
> For reference, here is a list of all closed JIRAs tagged with 3.1.0:
>
> https://s.apache.org/daffodil-issues-3.1.0
>
> For a summary of the changes in this release, see:
>
> https://daffodil.apache.org/releases/3.1.0/
>
> Please review and vote. The vote will be open for at least 72 hours
> (Saturday, 15 May 2021, 2pm EST).
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove (and reason why)
>

Re: Escape character parsing bug?

[Adams, Joshua]

So, after making the change to throw a Schema Definition Error whenever a 
terminator or separator begins with the escapeCharacter or 
escapeEscapeCharacter, around half of our escape scenario tests fail as they 
were all trying to test these weird edge cases for dealing with delimiters that 
start with the escapeCharacter or escapeEscapeCharacter.  I'm guessing that 
most of these tests can just be purged after a review to make sure we aren't 
losing coverage (other than this scenario where we are now throwing an SDE).  
Just wanted to get some opinions before moving forward with this change.

Josh
____
From: Adams, Joshua 
Sent: Tuesday, May 4, 2021 12:44 PM
To: dev@daffodil.apache.org 
Subject: Re: Escape character parsing bug?

I'll begin making the change to add an SDE for these then.  It seems that most 
of the escape scheme tests that weren't round tripping were cases like this.

Josh

On May 4, 2021 12:15 PM, "Beckerle, Mike"  wrote:
I asked Steve Hanson of IBM - other co-chair on DFDL workgroup, and one of the 
primaries on one of IBM's DFDL implementations, said that when he tries this 
situation with the escape character "/" matching the start of the separator, he 
gets an SDE.

It appears not to be part of the DFDL spec to call this out as an SDE, so that 
omission will likely become the first erratum to the DFDL v1.0 official final 
spec.


____
From: Adams, Joshua 
Sent: Monday, May 3, 2021 3:35 PM
To: dev@daffodil.apache.org 
Subject: Re: Escape character parsing bug?

Thanks for running this up the chain so to speak.  I agree that an SDE would 
probably be best for situations like this as I wouldn't think any sort of sane 
data format would use a combination of separators/escape characters like this.

Josh

From: Beckerle, Mike 
Sent: Monday, May 3, 2021 3:32 PM
To: dev@daffodil.apache.org 
Subject: Re: Escape character parsing bug?

So you have a separator the first char of which is the escape character.

Yikes. I think the DFDL spec should, ideally, make this an SDE. Feels entirely 
ambiguous to me.

The part of the spec you quote is quite problematic, but was updated by one 
word in the final DFDL Spec version.

Occurrences of the
dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed
from the data, unless the dfdl:escapeCharacter is preceded by the
dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter
does not precede the dfdl:escapeCharacter, respectively.

So breaking that into two independent statements:

  1.  An escapeCharacter is removed unless it is preceded by the escape-escape.
  2.  An escape-escape is removed unless it does not precede the escape 
character.

So (1) means an escape char that is floating around not in front of any 
delimiter is removed.
(2) means an escape-escape floating around not in front of any escape char, is 
preserved.

That still doesn't help with your specific issue. If a delimiter begins with 
the escapeCharacter, will that delimiter appearing in the data be interpreted 
as an escape character followed by the 2nd and subsequent characters of the 
delimiter? Or will the delimiter be recognized?

Consider dfdl:separator="/ // ///" with escapeCharacter="/" and 
escapeEscapeCharacter="/"

What takes priority, interpretation of escapeCharacters and 
escapeEscapeCharacters or recognizing delimiters?

I have posed this issue for consideration of the other DFDL workgroup members 
and I'll report back.


From: Adams, Joshua 
Sent: Monday, May 3, 2021 2:38 PM
To: dev@daffodil.apache.org 
Subject: Escape character parsing bug?

Consider the following schema:


  



  

  
  

  


We then have the following test case:
  

foo$$/;bar

  

  foo$/;bar

  

  

Shouldn't this parse as:

  foo$$
  bar


The spec says the following:
On parsing any in-scope terminating delimiter encountered in the data
is not interpreted as such when it is immediately preceded by the
dfdl:escapeCharacter (when not itself preceded by the
dfdl:escapeEscapeCharacter). Occurrences of the
dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed
from the data, unless the dfdl:escapeCharacter is preceded by the
dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter
does not precede the dfdl:escapeCharacter.

It seems to me that the '/;' terminator shouldn't be getting escaped in this 
case, but want to double check.

Josh

Re: Escape character parsing bug?

[Adams, Joshua]

I'll begin making the change to add an SDE for these then.  It seems that most 
of the escape scheme tests that weren't round tripping were cases like this.

Josh

On May 4, 2021 12:15 PM, "Beckerle, Mike"  wrote:
I asked Steve Hanson of IBM - other co-chair on DFDL workgroup, and one of the 
primaries on one of IBM's DFDL implementations, said that when he tries this 
situation with the escape character "/" matching the start of the separator, he 
gets an SDE.

It appears not to be part of the DFDL spec to call this out as an SDE, so that 
omission will likely become the first erratum to the DFDL v1.0 official final 
spec.


________
From: Adams, Joshua 
Sent: Monday, May 3, 2021 3:35 PM
To: dev@daffodil.apache.org 
Subject: Re: Escape character parsing bug?

Thanks for running this up the chain so to speak.  I agree that an SDE would 
probably be best for situations like this as I wouldn't think any sort of sane 
data format would use a combination of separators/escape characters like this.

Josh

From: Beckerle, Mike 
Sent: Monday, May 3, 2021 3:32 PM
To: dev@daffodil.apache.org 
Subject: Re: Escape character parsing bug?

So you have a separator the first char of which is the escape character.

Yikes. I think the DFDL spec should, ideally, make this an SDE. Feels entirely 
ambiguous to me.

The part of the spec you quote is quite problematic, but was updated by one 
word in the final DFDL Spec version.

Occurrences of the
dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed
from the data, unless the dfdl:escapeCharacter is preceded by the
dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter
does not precede the dfdl:escapeCharacter, respectively.

So breaking that into two independent statements:

  1.  An escapeCharacter is removed unless it is preceded by the escape-escape.
  2.  An escape-escape is removed unless it does not precede the escape 
character.

So (1) means an escape char that is floating around not in front of any 
delimiter is removed.
(2) means an escape-escape floating around not in front of any escape char, is 
preserved.

That still doesn't help with your specific issue. If a delimiter begins with 
the escapeCharacter, will that delimiter appearing in the data be interpreted 
as an escape character followed by the 2nd and subsequent characters of the 
delimiter? Or will the delimiter be recognized?

Consider dfdl:separator="/ // ///" with escapeCharacter="/" and 
escapeEscapeCharacter="/"

What takes priority, interpretation of escapeCharacters and 
escapeEscapeCharacters or recognizing delimiters?

I have posed this issue for consideration of the other DFDL workgroup members 
and I'll report back.


From: Adams, Joshua 
Sent: Monday, May 3, 2021 2:38 PM
To: dev@daffodil.apache.org 
Subject: Escape character parsing bug?

Consider the following schema:


  



  

  
  

  


We then have the following test case:
  

foo$$/;bar

  

  foo$/;bar

  

  

Shouldn't this parse as:

  foo$$
  bar


The spec says the following:
On parsing any in-scope terminating delimiter encountered in the data
is not interpreted as such when it is immediately preceded by the
dfdl:escapeCharacter (when not itself preceded by the
dfdl:escapeEscapeCharacter). Occurrences of the
dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed
from the data, unless the dfdl:escapeCharacter is preceded by the
dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter
does not precede the dfdl:escapeCharacter.

It seems to me that the '/;' terminator shouldn't be getting escaped in this 
case, but want to double check.

Josh

Re: Escape character parsing bug?

[Adams, Joshua]

Thanks for running this up the chain so to speak.  I agree that an SDE would 
probably be best for situations like this as I wouldn't think any sort of sane 
data format would use a combination of separators/escape characters like this.

Josh

From: Beckerle, Mike 
Sent: Monday, May 3, 2021 3:32 PM
To: dev@daffodil.apache.org 
Subject: Re: Escape character parsing bug?

So you have a separator the first char of which is the escape character.

Yikes. I think the DFDL spec should, ideally, make this an SDE. Feels entirely 
ambiguous to me.

The part of the spec you quote is quite problematic, but was updated by one 
word in the final DFDL Spec version.

Occurrences of the
dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed
from the data, unless the dfdl:escapeCharacter is preceded by the
dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter
does not precede the dfdl:escapeCharacter, respectively.

So breaking that into two independent statements:

  1.  An escapeCharacter is removed unless it is preceded by the escape-escape.
  2.  An escape-escape is removed unless it does not precede the escape 
character.

So (1) means an escape char that is floating around not in front of any 
delimiter is removed.
(2) means an escape-escape floating around not in front of any escape char, is 
preserved.

That still doesn't help with your specific issue. If a delimiter begins with 
the escapeCharacter, will that delimiter appearing in the data be interpreted 
as an escape character followed by the 2nd and subsequent characters of the 
delimiter? Or will the delimiter be recognized?

Consider dfdl:separator="/ // ///" with escapeCharacter="/" and 
escapeEscapeCharacter="/"

What takes priority, interpretation of escapeCharacters and 
escapeEscapeCharacters or recognizing delimiters?

I have posed this issue for consideration of the other DFDL workgroup members 
and I'll report back.

____
From: Adams, Joshua 
Sent: Monday, May 3, 2021 2:38 PM
To: dev@daffodil.apache.org 
Subject: Escape character parsing bug?

Consider the following schema:


  



  

  
  

  


We then have the following test case:
  

foo$$/;bar

  

  foo$/;bar

  

  

Shouldn't this parse as:

  foo$$
  bar


The spec says the following:
On parsing any in-scope terminating delimiter encountered in the data
is not interpreted as such when it is immediately preceded by the
dfdl:escapeCharacter (when not itself preceded by the
dfdl:escapeEscapeCharacter). Occurrences of the
dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed
from the data, unless the dfdl:escapeCharacter is preceded by the
dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter
does not precede the dfdl:escapeCharacter.

It seems to me that the '/;' terminator shouldn't be getting escaped in this 
case, but want to double check.

Josh

Escape character parsing bug?

[Adams, Joshua]

Consider the following schema:


  



  

  
  

  


We then have the following test case:
  

foo$$/;bar

  

  foo$/;bar

  

  

Shouldn't this parse as:

  foo$$
  bar


The spec says the following:
On parsing any in-scope terminating delimiter encountered in the data
is not interpreted as such when it is immediately preceded by the
dfdl:escapeCharacter (when not itself preceded by the
dfdl:escapeEscapeCharacter). Occurrences of the
dfdl:escapeCharacter and dfdl:escapeEscapeCharacter are removed
from the data, unless the dfdl:escapeCharacter is preceded by the
dfdl:escapeEscapeCharacter, or the dfdl:escapeEscapeCharacter
does not precede the dfdl:escapeCharacter.

It seems to me that the '/;' terminator shouldn't be getting escaped in this 
case, but want to double check.

Josh

Re: need 2nd reviewer on PR

[Adams, Joshua]

Sorry for the delay, I'll take a look at this tonight when I get home if it 
still needs a second review.

Josh

On Apr 29, 2021 12:42 PM, "Beckerle, Mike"  
wrote:
https://github.com/apache/daffodil/pull/539

Needs a second reviewer.

I added some "Highlights" comments to the files diffs to help you surf the 
deltas more effectively.

This fixes an issue a user was having doing streaming reads of messages.

Mike Beckerle | Principal Engineer

[cid:2f6f0715-6c4b-4778-8cfb-94509b2b116c]

mbecke...@owlcyberdefense.com

P +1-781-330-0412

Suspensions and NewVariableInstance

[Adams, Joshua]

I've been running into a lot of headaches trying to get newVariableInstance to 
correctly handle suspensions.  Currently when a newVariableInstance statement 
is found, a NewVariableInstanceStart and End unparsers are created. 
NewVariableInstanceStart will immediately create the newVariableInstance with 
no value and, if applicable, will create a SuspendableExpression that will 
calculate the default value. This works as expected but as the NVI's go out of 
scope we start running into some issues.

The NewVariableIsntanceEnd unparser simply removes the variable instance that 
was created in NVIStart.  It is not performing or checking for any sort of 
suspension, so NVIEnd is called while the NVIStart suspension is still active. 
Since the UStateForSuspension object uses the same VariableMap as the main 
UState object, this results in the variable's value not being correct after it 
goes out of scope.

I'm not sure what a good solution for this would be.  I've attempted adding a 
SuspendableOperation to the NVIEnd unparser to wait until the variable has a 
value before removing, but this results in a SuspensionDeadlock.  I've also 
attempted doing a deep copy of all the variables for each UStateForSuspension 
object, but this too results in a SuspensionDeadlock, not to mention that any 
changes made in one suspension wouldn't be visible to other suspensions.  One 
other thought I had, which I'm not even sure would even address the suspension 
issue, is to have whatever sequence is containing the NVI statement handle 
calling the NVIEnd unparser by simply adding it to the end of its sequence 
child unparsers.

Any thoughts on how best to handle this sticky situation of dealing with 
newVariableInstance and unparsing suspensions?

Re: [VOTE] Contributors - Graduate Apache Daffodil (Incubating) to a top-level project

[Adams, Joshua]

+1

We've seen a lot of new people contributing this past year.

Josh

On Jan 29, 2021 10:26 AM, "Sloane, Brandon"  wrote:
+1

-- Brandon

From: Mike Beckerle 
Sent: Thursday, January 28, 2021 6:08 PM
To: dev@daffodil.apache.org 
Subject: [VOTE] Contributors - Graduate Apache Daffodil (Incubating) to a 
top-level project

One of the first steps in graduating from the Apache incubator to top level
is making sure we have consensus among our contributors that this makes
sense now. This is an initial first step.

Please reply with your vote (+1, 0, or -1 with reasons)

This vote will be open for at least 72 hours (non-weekend), so until at
least 5pm US.ET (UTC-5) on Tuesday, Feb 2.

You can review our wiki page about how we meet the ASF project maturity
model which is something projects do to self-assess before moving forward.
Your comments on this wiki page are also welcome.

https://cwiki.apache.org/confluence/display/DAFFODIL/Apache+Daffodil+Maturity+Model+Assesment

About voting: see The Apache Voting Process:
https://www.apache.org/foundation/voting.html if you'd like to review the
process/guidance.

My vote +1.

NewVariableInstance Suspension Issue

[Adams, Joshua]

I'm working on wrapping up the variable direction feature and ran into a bug 
that is actaully unrelated to direction.  I have a test case that is pulled 
from the PCAP schema where on parse 4 elements are parsed, then a 5th element 
combines them to form an IP address with an IVC.


  
http://www.ogf.org/dfdl/";>
  

  
  
  

  http://www.ogf.org/dfdl/";>

  



  
http://www.ogf.org/dfdl/";>
  

  
  
  

  http://www.ogf.org/dfdl/";>

  


  

  
  


The issue I'm running into is on unparse I am attempting to use 
newVariableInstance to reverse the process by pulling apart the 5th IP address 
element.  What ends up happening is the expressions in the outputValueCalc's 
end up suspending, as expected, until the IPsrc element is unparsed. The 
problem is that while we are waiting for IPsrc to be unparsed, the 
newVariableInstance goes out of scope, which triggers the 
NewVariableInstanceEndUnparser and attempts to remove the variable instance, 
which hasn't been added yet as we were still waiting for the suspension to end.

I guess my question is would moving the NewVariableInstanceStart/EndUnparser's 
into a CombinatorUnparser fix this sort of suspension issue? I'm a little 
unfamiliar with the CombinatorUnparser, but there were comments in the code 
specifically saying that these NVIStart/End unparsers should really be 
implemented as a combinator unparser.  I'm guessing that since the NVI stuff 
would be wrapped into one parser it would suspend correctly.  If it becomes one 
unparser I'm a little unsure how the unparser would know when it has gone out 
of scope and call the appropriate clean up code.

Thoughts?

Josh

Re: Implementing variable "direction" property

[Adams, Joshua]

Good catch!  I had not implemented the SuspendableExpression stuff for 
NewVariableInstance.  It's going to take a little re-architecture as I had been 
treating expressions in newVariableInstance default values a little differently 
than expressions in setVariable, but I expect it will start working or at least 
uncover the next hurdle in getting this working.

As Steve said, this suspending can definitely cause some interesting things to 
happen with newVariableInstance, so I'll need to make sure I add proper test 
coverage.  If I'm lucky, I'll get this finished before I start Christmas 
vacation and forget everything, although my daycare situation implies I'm 
running low on luck, so we will see.

Josh

From: Beckerle, Mike 
Sent: Monday, December 21, 2020 10:25 AM
To: dev@daffodil.apache.org 
Subject: Re: Implementing variable "direction" property

Are you missing the use of a SuspendableExpression for NewVariableInstance?

I see SetVariableSuspendableExpression used in the Daffodil code base for 
SetVariable.

This suggests that the mechanism to suspend on a variable-reference that hits a 
variable the assignment of which is suspended will work.

(This seems to provide at least part of what Stevev L was pointing out as 
missing)

I see no equivalent for NewVariableInstance  in 
ExpressionEvaluatingUnparsers.scala









From: Steve Lawrence 
Sent: Monday, December 21, 2020 10:16 AM
To: dev@daffodil.apache.org 
Subject: Re: Implementing variable "direction" property

This is interesting. Normally, newVariableInstance/setVariable are not
allowed to have forward referencing expressions. This makes sense
because expression evaluated during parse must only be backwards
referencing. And normally the same newVariableInstance/setVariable
expressions are use during parse and unparse.

But with this new direction concept, unparseOnly expressions could
theoretically be allowed to be forward referencing, similar to how
outputValueCalc are allowed to be forward referencing. They are both
unparse only, so is fine.

But this means an unparseOnly NVI/SV expression evaluation must be
suspendable. And this means variables must have a concept of "I've been
set, but don't have a value yet because I'm waiting for some infoset
element to show up".

And this leads to a cascade of interesting behaviors. For example,
because our newVariableInstance must suspend, the OVC's that access that
variable must suspend when they try to use it but it has no value yet.
Fortnately, OVC's can already suspend, so I suspect this shouldn't be
too hard to add logic to suspend on "variable set but no value yet".

Things might also get tricky because variable state is mutable. When we
suspend a NVI/SV, we have to remember what instance was of that variable
we suspended at. And accesses to those (such as OVC) must also remember
what instance was accessed. I suspect this can all be made to work the
suspension and UState clones, but certainly complicates logic, and at
the very least might be worth considering if there is a different
approach to "direction" of variables.

On 12/21/20 9:54 AM, Adams, Joshua wrote:
> I've been working on DAFFODIL-2429, which is adding a "direction" property to 
> defineVariable. I believe I have most of the implementation in place. It is 
> primarily implemented as follows:
>
> dfdl:defineVariable has a property "dfdlx:direction" which is an enumeration 
> with the following values: "parseOnly", "unparseOnly", or "both" which is the 
> default value.
>
> When we are compiling the schema and are about to generate a SetVariabler or 
> NewVariableInstance parser/unparser, we check the "direction" property and if 
> the direction does not match (ie. Creating a SetVariable parser when the 
> variable in question is "unparseOnly") we instead create a NadaParser.
>
> I'm not 100% sure that this is necessarily the correct approach, but it 
> doesn't break any existing tests. Speaking of tests, I am attempting to 
> create a test to demonstrate this feature based off the pull request 
> mentioned in the bug ticket: https://github.com/DFDLSchemas/PCAP/pull/10
>
> The schema for my test is as follows:
>
>  dfdlx:direction="unparseOnly" />
> 
>   
> 
>   
> http://www.ogf.org/dfdl/";>
>defaultValue="{ ex:IPsrc }" />
> 
>   
>dfdl:outputValueCalc="{ 
> xs:unsignedByte(fn:substring-before($ex:remainingAddr, '.')) }" />
>   
> 
>   http://www.ogf.org/dfdl/";>
>   

Implementing variable "direction" property

[Adams, Joshua]

I've been working on DAFFODIL-2429, which is adding a "direction" property to 
defineVariable. I believe I have most of the implementation in place. It is 
primarily implemented as follows:

dfdl:defineVariable has a property "dfdlx:direction" which is an enumeration 
with the following values: "parseOnly", "unparseOnly", or "both" which is the 
default value.

When we are compiling the schema and are about to generate a SetVariabler or 
NewVariableInstance parser/unparser, we check the "direction" property and if 
the direction does not match (ie. Creating a SetVariable parser when the 
variable in question is "unparseOnly") we instead create a NadaParser.

I'm not 100% sure that this is necessarily the correct approach, but it doesn't 
break any existing tests. Speaking of tests, I am attempting to create a test 
to demonstrate this feature based off the pull request mentioned in the bug 
ticket: https://github.com/DFDLSchemas/PCAP/pull/10

The schema for my test is as follows:



  

  
http://www.ogf.org/dfdl/";>
  

  
  
  

  http://www.ogf.org/dfdl/";>

  



  
http://www.ogf.org/dfdl/";>
  

  
  
  

  http://www.ogf.org/dfdl/";>

  


  

  
  

  


This is pretty much a direct copy of the schema in the pull request that on 
parse will parse 4 bytes and then use an inputValueCalc to combine the 4 bytes 
into an IP address. On unparse outputValueCalc's is used to pull apart the 
combined address back into the individual bytes. During unparse, the byte* 
elements do a forward reference to IPsrc and this seems to be causing a problem 
in my test. I'm getting a "Unparse Error: Expression Evaluation Error: Child 
element {http://example.com}IPsrc does not exist".

So, my question regarding the test is should this work or am I missing 
something that is preventing this forward reference from working during 
unparsing?

Josh Adams

Re: [VOTE] Release Apache Daffodil (incubating) 3.0.0-rc1

[Adams, Joshua]

Voting again since my first vote got moderated (forgot to add my updated email 
address).

+1

Checked:

[OK]  "incubating" in name
[OK]  tgz and zip files contain the same files
[OK]  Verified all signatures
[OK]  Source builds and tests as expected
[OK]  No prebuilt binaries in src

Josh Adams

____
From: Adams, Joshua 
Sent: Monday, November 2, 2020 1:33 PM
To: dev@daffodil.apache.org 
Subject: Re: [VOTE] Release Apache Daffodil (incubating) 3.0.0-rc1

+1

Checked:

[OK]  "incubating" in name
[OK]  tgz and zip files contain the same files
[OK]  Verified all signatures
[OK]  Source builds and tests as expected
[OK]  No prebuilt binaries in src

Josh Adams


From: Steve Lawrence 
Sent: Friday, October 30, 2020 3:23 PM
To: dev@daffodil.apache.org 
Subject: [VOTE] Release Apache Daffodil (incubating) 3.0.0-rc1

Hi all,

I'd like to call a vote to release Apache Daffodil (incubating) 3.0.0-rc1.

All distribution packages, including signatures, digests, etc. can be
found at:

https://dist.apache.org/repos/dist/dev/incubator/daffodil/3.0.0-rc1/

Staging artifacts can be found at:

https://repository.apache.org/content/repositories/orgapachedaffodil-1019/

This release has been signed with PGP key 36F3494B033AE661,
corresponding to slawre...@apache.org, which is included in the KEYS
file here:

https://downloads.apache.org/incubator/daffodil/KEYS

The release candidate has been tagged in git with v3.0.0-rc1.

For reference, here is a list of all closed JIRAs tagged with 3.0.0:

https://s.apache.org/daffodil-issues-3.0.0

For a summary of the changes in this release, see:

https://daffodil.apache.org/releases/3.0.0/

Please review and vote. The vote will be open for at least 72 hours
(Monday, 2 November 2020, 3:30 EST).

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)

Thanks,
- Steve

Re: [VOTE] Release Apache Daffodil (incubating) 3.0.0-rc1

[Adams, Joshua]

+1

Checked:

[OK]  "incubating" in name
[OK]  tgz and zip files contain the same files
[OK]  Verified all signatures
[OK]  Source builds and tests as expected
[OK]  No prebuilt binaries in src

Josh Adams


From: Steve Lawrence 
Sent: Friday, October 30, 2020 3:23 PM
To: dev@daffodil.apache.org 
Subject: [VOTE] Release Apache Daffodil (incubating) 3.0.0-rc1

Hi all,

I'd like to call a vote to release Apache Daffodil (incubating) 3.0.0-rc1.

All distribution packages, including signatures, digests, etc. can be
found at:

https://dist.apache.org/repos/dist/dev/incubator/daffodil/3.0.0-rc1/

Staging artifacts can be found at:

https://repository.apache.org/content/repositories/orgapachedaffodil-1019/

This release has been signed with PGP key 36F3494B033AE661,
corresponding to slawre...@apache.org, which is included in the KEYS
file here:

https://downloads.apache.org/incubator/daffodil/KEYS

The release candidate has been tagged in git with v3.0.0-rc1.

For reference, here is a list of all closed JIRAs tagged with 3.0.0:

https://s.apache.org/daffodil-issues-3.0.0

For a summary of the changes in this release, see:

https://daffodil.apache.org/releases/3.0.0/

Please review and vote. The vote will be open for at least 72 hours
(Monday, 2 November 2020, 3:30 EST).

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)

Thanks,
- Steve

Problems with expressions and variables

[Adams, Joshua]

After spending a while banging my head against a wall as to why my simple test 
demonstrating newVariableInstance with a default value expression wasn't 
working, I think I understand what is going wrong now.

Here is a copy of the simple test case for newVariableInstance with a default 
value expression:


  

  
  

  

  http://www.ogf.org/dfdl/";>

  


  

  

  


I had added code to the VariableMap1 readVariable function to evaluate the 
expression if the variable didn't already have a value, so it was getting 
evaluated when element 'varValue' was being parsed.  This expression evaluation 
then triggered the evaluation of the myVar1, which itself is an expression that 
then needs to be evaluated.  The problem is when Daffodil attempts to run the 
expression for the newVariableInstance defaultValue, it does so from the 
context of 'varValue' and ultimately fails.

To me this seems like a problem that could potentially affect any expression 
that then references another non-constant expression.  I took a quick look 
through our tests and I was unable to find a test case that covered this 
scenario.  Is this something that Daffodil should support?  I took a look 
through the spec and didn't find anything prohibiting this, and it seems like 
my test case would describe the typical use case for newVariableInstance 
referencing a non-constant expression.

Assuming this test is a valid use case for a default value as an expression, 
the issue becomes when should the expression be evaluated?  This also could 
have issues that require some changes to the variable API.  My initial approach 
I described earlier attempted to evaluate the expression when the variable was 
read, but that resulted in an incorrect context for evaluating the variable's 
expression.  Another option would be to evaluate the expression when the 
VariableRuntimeData structure is created for the myVar1's newVariableInstance.  
This is the location where values are set for constant expressions, but the 
VariableRuntimeData class doesn't seem to have access to the PState or UState 
needed to evaluate non-constant expressions.

Just want to make sure that this is a valid test case and if so, what approach 
should be taken to get this working.

Josh

Re: [VOTE] Release Apache Daffodil (incubating) 2.7.0-rc2

[Adams, Joshua]

+1

Verified the following:

  *incubating in the name
  *   source compilation, unit tests and schema project tests all pass
  *   checked all hashes and signatures
  *   DISCLAIMER, NOTICE, LICENSE are all present

Josh Adams

From: Thompson, Dave 
Sent: Wednesday, July 1, 2020 10:27 AM
To: dev@daffodil.apache.org 
Subject: RE: [VOTE] Release Apache Daffodil (incubating) 2.7.0-rc2

+1

Verified/closed all resolved all v2.7.0  JIRA tickets.
Verified all incubator-daffodil sbt test suites execute successfully.
Verified all nightly test schemas compile and save successfully.
Verified the nightly test suite executes successfully.
Verified all dfdl-schemas repo sbt tests execute successfully.



-Original Message-
From: Mike Beckerle
Sent: Monday, June 29, 2020 1:40 PM
To: dev@daffodil.apache.org
Subject: [VOTE] Release Apache Daffodil (incubating) 2.7.0-rc2

Hi all,

I'd like to call a vote to release Apache Daffodil (incubating) 2.7.0-rc2.

All distribution packages, including signatures, digests, etc. can be found at:

https://dist.apache.org/repos/dist/dev/incubator/daffodil/2.7.0-rc2/


Staging artifacts can be found at:

https://repository.apache.org/content/repositories/orgapachedaffodil-1018/


This release has been signed with PGP key 13A680AF, corresponding to 
mbecke...@apache.org, which is included in the KEYS file here:

https://downloads.apache.org/incubator/daffodil/KEYS

The release candidate has been tagged in git with v2.7.0-rc2.

For reference, here is a list of all closed JIRAs tagged with 2.7.0:

https://s.apache.org/daffodil-issues-2.7.0

For a summary of the changes in this release, see:

https://daffodil.apache.org/releases/2.7.0/

Please review and vote. The vote will be open for at least 72 hours (ends on 
Thursday 2pm US.EDT = UTC-4)

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)

Please also mention in your mail any checking you performed on the release.

Thanks,
- Mike

Re: [DRAFT] Daffodil Podling Report - June 2020

[Adams, Joshua]

+1 from me as well.

Josh

From: Beckerle, Mike 
Sent: Thursday, May 28, 2020 10:58 AM
To: dev@daffodil.apache.org 
Subject: Re: [DRAFT] Daffodil Podling Report - June 2020

+1  Looks right to me.

Under community - if you want, you can add that Nteligen has two student summer 
interns starting Daffodil-related projects next week.

Or we can wait until the next report. Either way.





From: Steve Lawrence 
Sent: Thursday, May 28, 2020 9:43 AM
To: dev@daffodil.apache.org 
Subject: [DRAFT] Daffodil Podling Report - June 2020

## Daffodil

Apache Daffodil is an implementation of the Data Format Description
Language (DFDL) used to convert between fixed format data and XML/JSON.

Daffodil has been incubating since 2017-08-27.

### Three most important unfinished issues to address before graduating:

  1. Increase community growth and participation beyond Owl (formerly
 Tresys) (main priority)
  2. Work with other Apache projects where Daffodil could provide extra
 functionality
  3. Continue frequent release schedule

### Are there any issues that the IPMC or ASF Board need to be aware of?

  - None

### How has the community developed since the last report?

  - Pull requests merged from three first-time non-Owl contributors
  - Continued involvement in the mailing lists
  - dev: 78 postings, 5 new subscribers
  - users: 96 postings, 4 new subscribers
  - Owl is actively encouraging other companies with known Daffodil
interests to make contributions and increase involvement.
Anticipating initial on-list activity to begin in the coming weeks
  - Daffodil integrated into Smooks (https://www.smooks.org) by outside
developers

### How has the project developed since the last report?

  - Released Daffodil 2.6.0
  - 56 commits merged from 7 different contributors
  - Changes include thread-safe API improvements, cleanups based on
SonarCloud scans, improved validation of of input XML,
newVariableInstance feature, and many important misc bug fixes
  - 59 issues created, 61 issues resolved
  - Initial discussions and planning/ramping up on the proposed
code-generation runtime.
  - New proposals and plans in place to improve Daffodil streaming
capabilities and memory requirement reduction--main focus for the
next release

### How would you assess the podling's maturity?
Please feel free to add your own commentary.

  - [ ] Initial setup
  - [ ] Working towards first release
  - [X] Community building
  - [ ] Nearing graduation
  - [ ] Other:

### Date of last release:

  2020-04-24

### When were the last committers or PPMC members elected?

  - 2019-11-26 - Olabusayo Kilo (Committer)
  - 2019-06-20 - Brandon Sloane (PPMC)

### Have your mentors been helpful and responsive?

  - Yes

### Is the PPMC managing the podling's brand / trademarks?

  - No known cases of a 3rd party incorrectly using the Daffodil
name/brand.
  - Podling name search has been completed and approved by Brand
Management Committee:
https://issues.apache.org/jira/browse/PODLINGNAMESEARCH-147

Re: [VOTE] Release Apache Daffodil (incubating) 2.6.0-rc2

[Adams, Joshua]

+1

I checked:

Incubating in the name
Hashes and signature all correct
Source and Binaries include correct LICENSE, NOTICE, and DISCLAIMER
Source compiles and all tests pass
rpm installs and runs

Josh

From: Olabusayo Kilo 
Sent: Thursday, April 9, 2020 1:05 PM
To: dev@daffodil.apache.org 
Subject: [VOTE] Release Apache Daffodil (incubating) 2.6.0-rc2

Hi all,

I'd like to call a vote to release Apache Daffodil (incubating) 2.6.0-rc2.

All distribution packages, including signatures, digests, etc. can be
found at:

https://dist.apache.org/repos/dist/dev/incubator/daffodil/2.6.0-rc2/

Staging artifacts can be found at:

https://repository.apache.org/content/repositories/orgapachedaffodil-1015

This release has been signed with PGP key 639637FDA8049411,
corresponding to olabus...@apache.org, which is included in the KEYS
file here:

https://downloads.apache.org/incubator/daffodil/KEYS

The release candidate has been tagged in git with v2.6.0-rc2.

For reference, here is a list of all closed JIRAs tagged with 2.6.0:

https://s.apache.org/daffodil-issues-2.6.0

For a summary of the changes in this release, see:

https://daffodil.apache.org/releases/2.6.0/

Please review and vote. The vote will be open for at least 72 hours
(ends on Sunday, 12 April 2020, 12 Noon EST).

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)
--
Best Regards
Lola K.

Re: [DISCUSS] Daffodil 2.6.0 Release

[Adams, Joshua]

I did have interest, but it doesn't look like I'll have the time to do any 
Daffodil work for at least the next 2 weeks.  I have non-issue with someone 
else taking this one.

Josh

On Mar 25, 2020 3:56 PM, Olabusayo Kilo  wrote:
I'd like to have the below done for 2.6.0

https://issues.apache.org/jira/browse/DAFFODIL-2305 - Remove some unused
methods from Term.scala

https://issues.apache.org/jira/browse/DAFFODIL-2275 - Reduce code smells
and make some configuration changes to Sonarqube

I don't mind managing the release, although I think Josh A had expressed
interest last time.


On 3/25/20 3:24 PM, Beckerle, Mike wrote:
> I have two issues I'd like fixed for 2.6.0
>
> https://issues.apache.org/jira/browse/DAFFODIL-2302 - this is making it 
> difficult to update a Link16 schema that is in use by multiple parties.
>
> There's a second issue, which is that we still have this "NadaParsers are 
> supposed to optimize out" problem. Came up in this same schema again even 
> after the fix which supposedly addressed it. So I think the fix solved some, 
> but not all of this problem.  I will try to recreate this and reopen the 
> ticket if it is still happening.
>
> I don't want to manage the release, but I definitely think it's better if it 
> isn't Steve L., ... I could have my arm twisted.
> 
> From: Steve Lawrence 
> Sent: Wednesday, March 25, 2020 3:12 PM
> To: dev@daffodil.apache.org 
> Subject: [DISCUSS] Daffodil 2.6.0 Release
>
> With the last of the changes recently merged to support improved schema
> compilation speed and memory, I think now is a good time to start
> discussing the next release.
>
> 1) Does anyone have any changes that are close to being ready to merge
> or bugs that are worth delaying the next release until resolved?
>
> 2) Does anyone want to volunteer to be the release manager for 2.6.0?
> The release process [1] now uses containers for most of the heavy
> lifting, and since I did it last time it would be good to get someone
> else familiar with the new release process and make sure the container
> works for someone other than myself.
>
> - Steve
>
--
Best Regards
Lola K.

Re: [VOTE] Release Apache Daffodil (incubating) 2.5.0-rc2

[Adams, Joshua]

+1

Validated checksums for release packages.

From: Steve Lawrence 
Sent: Friday, January 3, 2020 11:29 AM
To: dev@daffodil.apache.org 
Subject: [VOTE] Release Apache Daffodil (incubating) 2.5.0-rc2

Hi all,

I'd like to call a vote to release Apache Daffodil (incubating)
2.5.0-rc2.

All distribution packages, including signatures, digests, etc. can be
found at:

https://dist.apache.org/repos/dist/dev/incubator/daffodil/2.5.0-rc2/

Staging artifacts can be found at:

https://repository.apache.org/content/repositories/orgapachedaffodil-1012/

This release has been signed with PGP key 36F3494B033AE661,
corresponding to slawre...@apache.org, which is included in the KEYS
file here:

https://www.apache.org/dist/incubator/daffodil/KEYS

The release candidate has been tagged in git with v2.5.0-rc2.

For reference, here is a list of all closed JIRAs tagged with 2.5.0:

https://s.apache.org/daffodil-issues-2.5.0

For a summary of the changes in this release, see:

https://daffodil.apache.org/releases/2.5.0/

Please review and vote. The vote will be open for at least 72 hours
(Monday 6 January, 2020, 11:30 AM EST).

[ ] +1 approve [ ] +0 no opinion [ ] -1 disapprove (and reason why)

Thanks,
- Steve

Re: [DISCUSS] Daffodil 2.5.0 Release

[Adams, Joshua]

+1

I'd be happy to take a crack at doing a release next time.

Josh

On Dec 20, 2019 7:39 AM, Steve Lawrence  wrote:
I think we're overdue for another release, and we have a lot of nice
features and bug fixes that people should start using. This includes
UDFs, BLOBs, unordered sequences, improved type safety, initial compiler
speed up changes, 2GB+ file support, and much more. And all non
work-in-progress pull requests have been merged.

The only outstanding issue I'm aware of is a potential performance
degradation in 2.5.0, but after some discussions it sounds like that is
likely related to the spectre/meltdown patches, and not recent code
changes. And we likely won't know for sure until after the new year.

So I suggest that we start the process to release 2.5.0 from the current
state of Daffodil, hopefully with the dev@daffodil.a.o and
general@incubator.a.o voting finishing and releasing sometime in early
January. If it turns out there is a legitimate performance degradation,
we can fix that and issue a point release.

Any thoughts/objections? If not, I'll volunteer to be the release
manager and start the process on Monday.

- Steve

Re: [VOTE] Release Apache Daffodil (incubating) 2.4.0-rc1

[Adams, Joshua]

+1

I checked the following:
 - incubating in name
 - signatures and hashes are all correct
 - DISCLAIMER exists
 - NOTICE looks correct
 - LICENSE looks correct

Josh

From: Steve Lawrence 
Sent: Wednesday, July 3, 2019 9:03 AM
To: dev@daffodil.apache.org
Subject: [VOTE] Release Apache Daffodil (incubating) 2.4.0-rc1

Hi all,

I'd like to call a vote to release Apache Daffodil (incubating) 2.4.0-rc1.

All distribution packages, including signatures, digests, etc. can be
found at:

https://dist.apache.org/repos/dist/dev/incubator/daffodil/2.4.0-rc1/

Staging artifacts can be found at:

https://repository.apache.org/content/repositories/orgapachedaffodil-1009/

This release has been signed with PGP key 36F3494B033AE661,
corresponding to slawre...@apache.org, which is included in the KEYS
file here:

https://www.apache.org/dist/incubator/daffodil/KEYS

The release candidate has been tagged in git with v2.4.0-rc1.

For reference, here is a list of all closed JIRAs tagged with 2.4.0:

https://issues.apache.org/jira/browse/DAFFODIL-1477?jql=project%20%3D%20DAFFODIL%20AND%20fixVersion%20%3D%202.4.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC

For a summary of the changes in this release, see:

https://daffodil.apache.org/releases/2.4.0/

Please review and vote. The vote will be open for at least 72 hours
(Saturday, 6 July 2019, 9am EST).

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)

Thanks,
- Steve

Re: [VOTE] Release Apache Daffodil (incubating) 2.4.0-rc1

[Adams, Joshua]

That one looks right to me.

Josh

From: Steve Lawrence 
Sent: Wednesday, July 3, 2019 10:23 AM
To: dev@daffodil.apache.org
Subject: Re: [VOTE] Release Apache Daffodil (incubating) 2.4.0-rc1

Both links look open up the same thing to me. I wonder if it's getting
mangled due to the length? Also, your link looks like it requires a log-in.

I've created an apache short url, does this look correct:

https://s.apache.org/daffodil-issues-2.4.0


On 7/3/19 10:15 AM, Beckerle, Mike wrote:
> The link for the list of tickets isn't correct. It opens up one specific 
> ticket for me. The correct URL
>
> is
>
> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20%3D%20DAFFODIL%20AND%20fixVersion%20%3D%202.4.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
>
>
> 
> From: Steve Lawrence 
> Sent: Wednesday, July 3, 2019 9:03:45 AM
> To: dev@daffodil.apache.org
> Subject: [VOTE] Release Apache Daffodil (incubating) 2.4.0-rc1
>
> Hi all,
>
> I'd like to call a vote to release Apache Daffodil (incubating) 2.4.0-rc1.
>
> All distribution packages, including signatures, digests, etc. can be
> found at:
>
> https://dist.apache.org/repos/dist/dev/incubator/daffodil/2.4.0-rc1/
>
> Staging artifacts can be found at:
>
> https://repository.apache.org/content/repositories/orgapachedaffodil-1009/
>
> This release has been signed with PGP key 36F3494B033AE661,
> corresponding to slawre...@apache.org, which is included in the KEYS
> file here:
>
> https://www.apache.org/dist/incubator/daffodil/KEYS
>
> The release candidate has been tagged in git with v2.4.0-rc1.
>
> For reference, here is a list of all closed JIRAs tagged with 2.4.0:
>
> https://issues.apache.org/jira/browse/DAFFODIL-1477?jql=project%20%3D%20DAFFODIL%20AND%20fixVersion%20%3D%202.4.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
>
> For a summary of the changes in this release, see:
>
> https://daffodil.apache.org/releases/2.4.0/
>
> Please review and vote. The vote will be open for at least 72 hours
> (Saturday, 6 July 2019, 9am EST).
>
> [ ] +1 approve
> [ ] +0 no opinion
> [ ] -1 disapprove (and reason why)
>
> Thanks,
> - Steve
>

choiceLengthKind='explicit'

[Adams, Joshua]

Wanted to make sure I am understanding the spec for choiceLengthKind correctly:

 - 'explicit' means that the branches of the choice are always filled to the 
fixed
length specified by dfdl:choiceLength, so the ChoiceContent region is fixed
length regardless of which branch appears.

And under choiceLength:

 - Specifies the length of the choice in bytes, so the ChoiceContent region is
fixed length regardless of which branch appears. A ChoiceUnused region
is therefore possible which when unparsing is filled with dfdl:fillByte.

So, essentially this is saying that a choice with choiceLengthKind='explicit' 
has no branches that will exceed the length specified in choiceLength, correct? 
 And, that on unparse, any branch that is less than choiceLength will simply be 
padded by dfdl:fillByte.  Assuming that is correct, it shouldn't take much work 
to get this working in Daffodil.

That being said, is there any sample data around? I searched through the 
daffodil repo as well as some of the schema projects and did not find any.

Josh

Schema Definition Error vs Processing Error for empty choiceDispatchKey

[Adams, Joshua]

In working on DAFFODIL-2083 with empty string in delimiters, I made some 
changes that also effect choiceDispatchKey.  For delimiters it is a Schema 
Definition Error if an expression evaluates to an empty string.  We had 
originally been creating a Parse Error (although it was getting masked by an 
Assert check) and in changing it to produce an SDE it changed the error type 
for choiceDispatchKey.
The spec for choiceDispatchKey says "The expression must evaluate to an 
xs:string which must not be the empty string." but it doesn't say what kind of 
error an empty string should produce.  It later says that if the 
choiceDispatchkey does not one of the choiceBranchKeys it is a processing 
error.  So is it correct to do a processing error for a choiceDispatchKey that 
evaluates to an empty string?

Josh

Re: [VOTE] Release Apache Daffodil (incubating) 2.3.0-rc1

[Adams, Joshua]

+1


From: Steve Lawrence 
Sent: Thursday, February 14, 2019 1:11:13 PM
To: dev@daffodil.apache.org
Subject: [VOTE] Release Apache Daffodil (incubating) 2.3.0-rc1

Hi all,

I'd like to call a vote to release Apache Daffodil (incubating) 2.3.0-rc1.

All distribution packages, including signatures, digests, etc. can be
found at:

https://dist.apache.org/repos/dist/dev/incubator/daffodil/2.3.0-rc1/

Staging artifacts can be found at:

https://repository.apache.org/content/repositories/orgapachedaffodil-1008/

This release has been signed with PGP key 36F3494B033AE661,
corresponding to slawre...@apache.org, which is included in the
repository's KEYS file. This key can be found on keyservers, such as:

http://pgp.mit.edu/pks/lookup?op=get&search=0x36F3494B033AE661

It is also listed here:

https://people.apache.org/keys/committer/slawrence.asc

The release candidate has been tagged in git with v2.3.0-rc1.

For reference, here is a list of all closed JIRAs tagged with 2.3.0:

https://issues.apache.org/jira/browse/DAFFODIL-2067?jql=project%20%3D%20DAFFODIL%20AND%20status%20%3D%20Resolved%20AND%20fixVersion%20%3D%202.3.0%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC

For a summary of the changes in this release, see:

https://daffodil.apache.org/releases/2.3.0/

Please review and vote. The vote will be open for at least 72 hours.

[ ] +1 approve
[ ] +0 no opinion
[ ] -1 disapprove (and reason why)

Thanks,
- Steve