[jira] [Closed] (DAFFODIL-1978) Change runtime to use vectors not lists/seq
[ https://issues.apache.org/jira/browse/DAFFODIL-1978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1978. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 8dcb4e4cc8f486bb5ef082719be36ef519fd361e. Verified daffodil successfully builds and executes all sbt tests. Verified changes specified in the commit comment by code review. All nightly tests also executed successfully. > Change runtime to use vectors not lists/seq > --- > > Key: DAFFODIL-1978 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1978 > Project: Daffodil > Issue Type: Improvement > Components: Clean Ups >Affects Versions: 2.1.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Minor > Fix For: 2.2.0 > > > The runtime should be using Vector, not Seq/List. > Change childProcessors and runtimeDependencies to Vector types, and all > usages then as well. > The core/compiler primtives layer is the bridge between DSOM and the grammar, > which deal in List/Seq, and the runtime, which uses Vectors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1960) RepUnboundedParser incorrectly handling points of uncertainty
[ https://issues.apache.org/jira/browse/DAFFODIL-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1960. --- Closing as duplicate to DAFFODIL-1985 per dev team. DAFFODIL-1985 has been verified and closed. > RepUnboundedParser incorrectly handling points of uncertainty > - > > Key: DAFFODIL-1960 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1960 > Project: Daffodil > Issue Type: Improvement > Components: Back End >Affects Versions: 2.1.0 >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.2.0 > > > The RepUnboundedParser currently creates two marks: startState and > priorState. The priorState mark changes as we go through each repetition of > the array. The startState mark never changes. If an error occurs during a > repetition and a discriminator *IS NOT* set, we reset to the prior mark, > effectively backtracking the single failed repetition and keep all successful > repetitions of the array. However, if an error occurs during a repetition and > a discriminator *IS* set, then we backtrack the entire array back to the > startState mark. > This behavior is not correct. This behavior treats the beginning of an array > as if it were a point of uncertainty, but that is not the case. Only each > element of the array is a point of uncertainty. So there is no need for the > startState mark. It should be removed, and if a repetition fails and a > discriminator is set then we should simply discard priorState mark and > backtrack to whatever point of uncertainty enclosed the array (i.e. just > return from the parse call and let containing parsers handle it). > We should also examine the other repetition parsers and ensure they have the > correct behavior. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1943) Date, Time, DateTime - Binary - binaryCalendarRep='ibm4690Packed'
[ https://issues.apache.org/jira/browse/DAFFODIL-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1943. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 3877a44c790b562b6c0799abae815e35a105ebd0. Verified daffodil builds and executes all sbt tests successfully, including the test cases added by the commit. Verified changes specified in the commit comment. All nightly tests also executed successfully. > Date, Time, DateTime - Binary - binaryCalendarRep='ibm4690Packed' > - > > Key: DAFFODIL-1943 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1943 > Project: Daffodil > Issue Type: New Feature > Components: DFDL Language, Front End >Reporter: Elizabeth Fahl >Assignee: Elizabeth Fahl >Priority: Major > Fix For: 2.2.0 > > > Add the value 'ibm4690Packed' for 'binaryCalendarRep' property. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1942) Date, Time, DateTime - Binary - binaryCalendarRep='packed'
[ https://issues.apache.org/jira/browse/DAFFODIL-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1942. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 8a694b5fd57265a0457ae94c2adc0d21f476a4e6. Verified daffodil builds and executes all sbt tests successfully, including the test cases added and by the commit. Verified changes specified in the commit comment. All nightly tests also executed successfully. > Date, Time, DateTime - Binary - binaryCalendarRep='packed' > -- > > Key: DAFFODIL-1942 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1942 > Project: Daffodil > Issue Type: New Feature > Components: DFDL Language, Front End >Reporter: Elizabeth Fahl >Assignee: Elizabeth Fahl >Priority: Major > Fix For: 2.2.0 > > > Add the value 'packed' for 'binaryCalendarRep' property. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1941) Date, Time, DateTime - Binary - binaryCalendarRep='bcd'
[ https://issues.apache.org/jira/browse/DAFFODIL-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1941. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 339967faf80ca9b13f8a3725032e79b5c988a8fd. Verified daffodil builds and executes all sbt tests successfully, including the test cases added and by the commit. Verified changes specified in the commit comment. All nightly tests also executed successfully. > Date, Time, DateTime - Binary - binaryCalendarRep='bcd' > --- > > Key: DAFFODIL-1941 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1941 > Project: Daffodil > Issue Type: New Feature > Components: DFDL Language, Front End >Reporter: Elizabeth Fahl >Assignee: Dave Thompson >Priority: Major > Fix For: 2.2.0 > > > Add the value 'bcd' for 'binaryCalendarRep' property. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1992) Generated scaladoc has very long links that are hard to read
[ https://issues.apache.org/jira/browse/DAFFODIL-1992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1992. --- Pulled latest updates from incubator-daffodil repository which included specified commit, b04e86d9cc3f72dc13b5c051423e3d78a9244edf. Generated the scaladocs. Verified scaladocs links are reasonable in length and easier to read. > Generated scaladoc has very long links that are hard to read > > > Key: DAFFODIL-1992 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1992 > Project: Daffodil > Issue Type: Bug > Components: Documentation >Affects Versions: 2.2.0 >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.2.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1991) lengthPattern="[\x00-\xFF]{0,80}" will only match up to 64 bytes
[
https://issues.apache.org/jira/browse/DAFFODIL-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Thompson closed DAFFODIL-1991.
---
Pulled latest updates from incubator-daffodil repository which included
specified commit, 6b5383020979077ce0c7360fa42654e206a2c3c2.
Verified daffodil builds and executes all sbt tests successfully, including the
associated test case added by the specified commit.
Verified changes specified in the commit comment.
All nightly tests also executed successfully.
> lengthPattern="[\x00-\xFF]{0,80}" will only match up to 64 bytes
>
>
> Key: DAFFODIL-1991
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1991
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 2.2.0
>Reporter: Steve Lawrence
>Assignee: Steve Lawrence
>Priority: Major
> Fix For: 2.2.0
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1930) DFDLGeneralFormat defines calendarTimeZone as UTC - should define as ""
[ https://issues.apache.org/jira/browse/DAFFODIL-1930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1930. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 4522a82d9739a4e47cc3dd89af76e05f54abcd08. Verified daffodil builds and executes all sbt tests successfully, including the test case modified by the commit. Verified changes specified in the commit comment included. All nightly tests also executed successfully. > DFDLGeneralFormat defines calendarTimeZone as UTC - should define as "" > --- > > Key: DAFFODIL-1930 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1930 > Project: Daffodil > Issue Type: Bug > Components: Front End >Affects Versions: 2.1.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.2.0 > > > Because DFDLGeneralFormat.dfdl.xsd defines calendarTimeZone="UTC" dates like > "7/4/1999" come through in the infoset as 1999-07-04+00:00, that is, with the > trailing time-zone specifier. > The right definition for calendarTimeZone for a general purpose format is "", > which means "time zone unknown". > This avoids the need for all users of DFDLGeneralFormat who have dates to > override calendarTimeZone themselves to "" value, and puts the burden on them > to define it if they know and want a default time zone. > Note: changing this will likely change the behavior of dateTime type as well > as the date type, and we should not be taking time zone information or > defaults from the local machine settings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1987) Buckets not released with small bucket sizes
[ https://issues.apache.org/jira/browse/DAFFODIL-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1987. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 12ad0019bfb2feb7a7c713743f40865bd00795aa. Verified daffodil builds and executes all sbt tests successfully. Verified changes specified in the commit comment. All nightly tests also executed successfully. > Buckets not released with small bucket sizes > > > Key: DAFFODIL-1987 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1987 > Project: Daffodil > Issue Type: Bug > Components: Back End >Affects Versions: 2.2.0 >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.2.0 > > > Setting the bucket size to something small prevents any buckets from being > released. Need to investigate and fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1799) Enable data streaming in the CLI
[ https://issues.apache.org/jira/browse/DAFFODIL-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1799. --- Closed ticket as duplicate per dev team (1968). > Enable data streaming in the CLI > > > Key: DAFFODIL-1799 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1799 > Project: Daffodil > Issue Type: Bug > Components: CLI, Performance >Reporter: Steve Lawrence >Priority: Major > Fix For: 2.2.0 > > > This CLI currently reads the input data as a byte array. This is simple and > allows for ensuring all data is read into a memory, reducing disk overhead > during the preformance command. However, this means the CLI is limited to the > maximum size of an array, which is INT_MAX. In order to support the CLI > parsing/unparsing larger files, we should instead work on InputStreams rather > than array buffers. For the performance subcommand, this will mean requiring > something like a SplittalbeInputStream that will allow multiple consumers of > a single InputStream. > Some SplittableInputStream implementations do exist, for example in JMRTD and > on stack overflow, but licensing issues make it so these aren't an option. > Either need to find a solution compatible with our license or implement our > own. > This work should be done concurrently with changes to improve the efficiency > of the I/O layer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1882) Implement support for the various formats of binaryCalendarRep
[ https://issues.apache.org/jira/browse/DAFFODIL-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1882. --- Closing as duplicate per dev team. > Implement support for the various formats of binaryCalendarRep > -- > > Key: DAFFODIL-1882 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1882 > Project: Daffodil > Issue Type: Bug > Components: Compatibility >Affects Versions: 2.0.0 >Reporter: Josh Adams >Priority: Major > Fix For: 2.2.0 > > > Implement support for the various formats of binaryCalendarRep: > * binarySeconds > * binaryMilliseconds > * packed > * bcd > * ibm4690Packed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1947) TDMLRunner - incorrect error on left-over data: TDMLException: Left over data. Consumed 48 bit(s) with 0 bit(s) remaining.
[ https://issues.apache.org/jira/browse/DAFFODIL-1947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1947. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 383148eaeebe57879af2e7589aa1b08e08280ebd. Verified daffodil builds and executes all sbt tests successfully, including the test case added and modified (twopass, threepass) by the commit. Verified changes specified in the commit comment. All nightly tests also executed successfully. > TDMLRunner - incorrect error on left-over data: TDMLException: Left over > data. Consumed 48 bit(s) with 0 bit(s) remaining. > -- > > Key: DAFFODIL-1947 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1947 > Project: Daffodil > Issue Type: Bug > Components: TDML Runner >Affects Versions: 2.1.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.2.0 > > > When a test runs round trip, if the test parse succeeds, but the unparse > creates an extra character in the output, on reparse we get a left-over-data > error, but the error message is bogus because it says 0 bits remaining, which > makes no sense for left-over data. > org.apache.daffodil.tdml.TDMLException: Left over data. Consumed 48 bit(s) > with 0 bit(s) remaining. > In general, the round-trip logic should be a bit smarter. The output should > indicate what phase of the test it was in when the failure occurred. In this > specific case, the unparsed output doesn't match the size of the original > input. That might be ok on round trip, but in most cases we want > parse/unparse to work without having to reparse. > We need to fix the bug so that this doesn't report incorrect left-over-data > info. We also need to change roundTrip to have a mode for one-pass only, > meaning that the unparse is expected to create exactly the same as the input > data (for parse tests), so that it should not try to reparse the unparsed > output, but should immediately report the error. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1939) Errata 5.32 to 5.38 status is not on the Unimplemented Features/Errata page
[
https://issues.apache.org/jira/browse/DAFFODIL-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Thompson closed DAFFODIL-1939.
---
{color:#33}Verified the
[{color:#0066cc}https://daffodil.apache.org/unsupported/{color}] site now lists
the{color} entries for errata 5.32 to 5.38.
> Errata 5.32 to 5.38 status is not on the Unimplemented Features/Errata page
> ---
>
> Key: DAFFODIL-1939
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1939
> Project: Daffodil
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 2.1.0
>Reporter: Michael Beckerle
>Assignee: Michael Beckerle
>Priority: Minor
> Fix For: 2.2.0
>
>
> The page in the Apache Daffodil site
> https://daffodil.apache.org/unsupported/
> that lists the unimplemented features and status w.r.t. the errata, that page
> is missing entries for errata 5.32 to 5.38.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1519) occursCount that is greater than maxOccursBounds causes PE/UE rather than SDE
[ https://issues.apache.org/jira/browse/DAFFODIL-1519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1519. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 62da5b5a44e3f90c42ed446f7e64950ff61b6738. Verified daffodil builds and executes all sbt tests successfully, including the test case added by the commit. Verified changes specified in the commit comment included. All nightly tests also executed successfully. > occursCount that is greater than maxOccursBounds causes PE/UE rather than SDE > - > > Key: DAFFODIL-1519 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1519 > Project: Daffodil > Issue Type: Bug > Components: Back End, Diagnostics, Usability >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.2.0 > > > Currently, if you have an expression for occursCount, and that expression > results in a value greater than maxOccursBounds (which defaults to 1024), > then we currently throw a PE/UE. When parsing, this can result in > backtracking for a non-obvious reason. The fact that maxOccursBounds was > exceeded may not even be reported. > Instead, it seems like this should instead be a SDE so that the error is > reported immediately and the user can either modify a tunable to increase the > maxOccursBOunds, or rework the schema to not need such a large array. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-991) DFDL-Spec: Update spec to reflect change to Schema Definition Warning for alignment ambiguity
[ https://issues.apache.org/jira/browse/DAFFODIL-991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-991. -- Pulled latest updates from incubator-daffodil repository which included specified commit, ed14cb351bbba6c2f98ae18ae957db6cca448a8b. Verified daffodil builds and executes all sbt tests successfully, including the test cases specified in this JIRA ticket, with the exception of test alignmentOptionalElem03. An issue with this test is addressed in JIRA ticket 1217. Verified changes specified in the commit comment included. All nightly tests also executed successfully. > DFDL-Spec: Update spec to reflect change to Schema Definition Warning for > alignment ambiguity > - > > Key: DAFFODIL-991 > URL: https://issues.apache.org/jira/browse/DAFFODIL-991 > Project: Daffodil > Issue Type: Task > Components: DFDL Language, QA >Affects Versions: s14 >Reporter: Elizabeth Fahl >Assignee: Dave Thompson >Priority: Minor > Fix For: 2.2.0 > > > When parsing an optional element, alignment that differs from following > elements has been changed from a Schema Definition Error to a Schema > Definition Warning. The spec needs to be updated to reflect this. > This is in section 12.1 Aligned Data: > "To avoid ambiguity when parsing, optional elements where the minimum number > of occurrences is zero cannot have alignment properties different from the > items that follow them. It is a schema definition error otherwise." > The requirement "DFDL-12-011R:alignment - avoid ambiguity when parsing" > should be updated as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1984) Separator prefix of terminator - not getting longest match
[ https://issues.apache.org/jira/browse/DAFFODIL-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1984. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 73d071750b7b484e010cb357f226b93de2f4439a. Verified daffodil builds and executes all sbt tests. Verified changes specified in the commit comment by code review. All nightly tests also executed successfully. Successfully performed streaming tests on VMF data formats. > Separator prefix of terminator - not getting longest match > -- > > Key: DAFFODIL-1984 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1984 > Project: Daffodil > Issue Type: Bug > Components: Back End >Affects Versions: 2.2.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Blocker > Fix For: 2.2.0 > > > Regression in new separator/sequences code. > When the separator is a prefix of the terminator, or any other piece of > in-scope terminating markup, then the separator match is preferred to the > longest match, and that makes any format that depends on longest-match > disambiguation to fail. > The one example we have of this is in VMF's spec scraper. > So we also need a test of this in daffodil-test to have better coverage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1964) Unparsing ArrayCombinator assertion failure
[
https://issues.apache.org/jira/browse/DAFFODIL-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Thompson closed DAFFODIL-1964.
---
Pulled latest updates from incubator-daffodil repository which included
specified commit, 73d071750b7b484e010cb357f226b93de2f4439a.
Verified daffodil builds and executes all sbt tests successfully, including the
test cases specified in this JIRA ticket.
Verified changes specified in the commit comment included.
All nightly tests also executed successfully.
> Unparsing ArrayCombinator assertion failure
> ---
>
> Key: DAFFODIL-1964
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1964
> Project: Daffodil
> Issue Type: Bug
> Components: Back End, Unparsing
>Affects Versions: 2.1.0
>Reporter: Steve Lawrence
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.2.0
>
>
> A user reported an error when unparsing in the array combinator.
> The schema looks something like this:
> {code}
> ...
>
>
>dfdl:occursCountKind="implicit">
>
>
>
> http://www.ogf.org/dfdl/;>
> { expression }
>
>
>
>
>
>
>
>
> ...
> {code}
> With XML that looks like this:
> {code}
> ...
> {code}
> So the bar element is an array that requires at least one instance, but the
> infoset has zero instances. Trying to unparse this resultsin the following:
> {code}
> org.apache.daffodil.exceptions.Abort: Invariant broken:
> event.isStart.&&(event.node.isInstanceOf[org.apache.daffodil.infoset.DIArray])
> org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:129)
> org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:129)
> org.apache.daffodil.processors.unparsers.ArrayCombinatorUnparser.unparse(ElementKindUnparsers.scala:254)
> org.apache.daffodil.processors.unparsers.Unparser$class.unparse1(Unparser.scala:72)
> ...
> {code}
> The expected result is to return an UnaparseError about expecting a start
> array event but not getting one. I think the assertion just needs to be
> changed to an UnparseError, since there is clearly a case where the assertion
> doesn't hold.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1938) RPM/tar/zips contain jars with different hashes
[ https://issues.apache.org/jira/browse/DAFFODIL-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1938. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 80d40e01f533ffe04fa8fa6e12be7b73d4eff269. Verified changes specified in the commit comment by code review. Performed the following commands to create daffodil rpm, tar and zip files, respectively: * sbt daffodil-cli/rpm:packageBin * sbt daffodil-cli/universal:packageZipTarball * sbt daffodil-cli/universal:packageBin Extracted the jar files from each and performed md5sum checks on each set of jar files. Verified the md5sum hashes of jar files in each set were the same. > RPM/tar/zips contain jars with different hashes > --- > > Key: DAFFODIL-1938 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1938 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Critical > Fix For: 2.2.0 > > > When generating rpm, zip, and tar for release, the hashes of the daffodil > jars are different in the rpm than in the zip and tar. Since it is rebuild, > it appears the only changes are things like timestamps--the content is > exactly the same. However, this makes it difficult to verify that the jars > are the same during a release process. This appears to be a bug with the > sbt-native-packager plugin always rebuilding the packaged jars each time it > creates an rpm or zip. Issue has been opened on the sbt-native-packager bug > tracker to investigate this: > https://github.com/sbt/sbt-native-packager/issues/1130 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1980) suppressWarnings vs. suppressSchemaDefinitionWarnings in config files
[ https://issues.apache.org/jira/browse/DAFFODIL-1980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1980. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 2612ad8da23a70ebefd82f351ad83db8d8e89022. Verified daffodil builds and executes all sbt tests successfully. Verified changes specified in this JIRA ticket. All nightly tests also executed successfully. > suppressWarnings vs. suppressSchemaDefinitionWarnings in config files > - > > Key: DAFFODIL-1980 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1980 > Project: Daffodil > Issue Type: Bug > Components: API, Front End >Affects Versions: 2.1.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.2.0 > > > The code isn't consistent about whether the name for suppressing warnings is > "suppressWarnings" or "suppressSchemaDefinitionWarnings". This made it > impossible to use the feature. > Change to suppressSchemaDefinitionWarnings since that is the name of the > actual tunable. > Or if we like suppressWarnings better we should change the name uniformly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1961) TDML Runner - enhance round-trip to distinguish simple parse-unparse from multi-trip cases
[ https://issues.apache.org/jira/browse/DAFFODIL-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1961. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 383148eaeebe57879af2e7589aa1b08e08280ebd. Verified daffodil builds and executes all sbt tests successfully. Verified changes specified in the commit comment. All nightly tests also executed successfully. > TDML Runner - enhance round-trip to distinguish simple parse-unparse from > multi-trip cases > -- > > Key: DAFFODIL-1961 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1961 > Project: Daffodil > Issue Type: New Feature > Components: TDML Runner >Affects Versions: 2.1.0 >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.2.0 > > > TDML Runner does parse-unparse round-trip tests. > Some tests the input data is not cannonical, but after parse-unparse, the > data *IS* cannonical. Hence, a Parse-Unparse-Parse is required before the > Infoset matches. > However, very few tests require this. Those should be explicitly identified > by changing the TDML roundTrip='true/false' attribute into an enumeration: > roundTrip="false/true/simple/multiPass" > The values true and simple mean the same thing. The value multi-pass means > that a parse-unparse-parse cycle is needed for a parse test, and an > unparse-parse-unparse cycle is needed for an unparse test. > TBD: are longer cycles actually needed? If so then tests that require P-U-P-U > should be distinghished from those that require only P-U-P, perhaps by > changing the enums for round trip to "twoPass" and "threePass". > By specifying this more specific need for passes, the intent of the test > writer is clearer. > This also avoids cascading errors where a test that should not require > multiple passes, is failing, multiple-passes are attempted, and the failure > one observes is the multi-pass failure. This can just be some artifact, and > not have much useful value when debugging. You want the first failure, in the > case of a simple round-trip test. You don't want it to try multi-pass looping. > Right now I have 140-or-so failures in our TDML-based test suite > (daffodil-test). I'd very much like to know which of these are simple > failures, and debug those, and ignore the multi-pass failures for now, but I > have no way of distinguishing them currently. > This change could also improve test time - because re-running tests in the > multi-test loops wouldn't even be attempted for tests that don't require it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-851) textNumberProps: raw byte entity should not be allowed
[ https://issues.apache.org/jira/browse/DAFFODIL-851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-851. -- Pulled latest updates from incubator-daffodil repository. Verified daffodil builds and executes all sbt tests successfully, including the test cases specified in this JIRA ticket. All nightly tests also executed successfully. > textNumberProps: raw byte entity should not be allowed > -- > > Key: DAFFODIL-851 > URL: https://issues.apache.org/jira/browse/DAFFODIL-851 > Project: Daffodil > Issue Type: Bug > Components: Back End, General >Affects Versions: s11 >Reporter: Jessie Chab >Priority: Major > Fix For: 2.2.0 > > > According to the spec, raw byte entities should not be allowed for > textNumberStandardDecimalSeparator, textNumberStandardGroupingSeparator, etc. > I created a few tests for this, and the data parses without an issue. > textStandardGroupingSeparator12 > textStandardDecimalSeparator17 > in files: > daffodil-test/src/test/resources/edu/illinois/ncsa/daffodil/section13/text_number_props/TextNumberProps.tdml > daffodil-test/src/test/scala-debug/edu/illinois/ncsa/daffodil/section13/text_number_props/TestTextNumberPropsDebug.scala > daffodil-test/src/test/scala/edu/illinois/ncsa/daffodil/section13/text_number_props/TestTextNumberProps.scala -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1977) Misspellings in Java/Scala API doc
[ https://issues.apache.org/jira/browse/DAFFODIL-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1977. --- Pulled latest updates from incubator-daffodil repository which included specified commit, a19285ef738d1c0bd996083fa077909c0eb6a54f. Verified changes specified in the commit comment by code review. > Misspellings in Java/Scala API doc > -- > > Key: DAFFODIL-1977 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1977 > Project: Daffodil > Issue Type: Bug > Components: API, Documentation >Affects Versions: 2.1.0 >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Labels: beginner > Fix For: 2.2.0 > > > The word "recieve" is spelled incorrectly in the documentation for the Java > and Scala API documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1923) Excel export CSV escaping - can't be handled with standard block escape
[
https://issues.apache.org/jira/browse/DAFFODIL-1923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Thompson closed DAFFODIL-1923.
---
Pulled latest updates from incubator-daffodil repository which included
specified commit, 59654346802a1a16c3c1461f48a8d19397287044. Note: Merged commit
hash is a83b3772bc5d6eccb127babc7278ef797db29c75.
Verified daffodil builds and executes all sbt tests, including the associated
test cases added in the specified commit.
Verified changes specified in the commit comment included.
Verified all nightly tests execute successfully.
> Excel export CSV escaping - can't be handled with standard block escape
> ---
>
> Key: DAFFODIL-1923
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1923
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 2.1.0
>Reporter: Michael Beckerle
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.2.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Excel spreadsheet looks like this:
> |1|2|This is "some fun" now|3
> When exported to CSV, this is created
> {code}
> 1,2,"This is ""some fun"" now",3
> {code}
> Notice the quotations around the string, and the doubling of the quotes
> inside the string as an escaping mechanism.
> One would think that an escape scheme with kind 'escapeBlock' with the block
> start, block end, escape and escapeEscape all defined to be '"' (doublequote)
> would work, but it does not.
> Error message is a PE indicating "delimiter not found".
> I am only 60% sure of this bug. It came up in a project using CSV data
> exported from excel, but other complexities there may have been interacting.
> Isolated test case is needed to really verify the behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1979) UTF8 decoder doesn't handle 3-byte and 4-byte correctly
[ https://issues.apache.org/jira/browse/DAFFODIL-1979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1979. --- Pulled latest updates from incubator-daffodil repository which included specified commit, d50e1fa098feec407a8aae07921d1f1e885e4ff5. Verified daffodil builds and executes all sbt tests successfully, including the associated test case added in the specified commit. Verified changes specified in the commit comment. All nightly tests also executed successfully. > UTF8 decoder doesn't handle 3-byte and 4-byte correctly > --- > > Key: DAFFODIL-1979 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1979 > Project: Daffodil > Issue Type: Bug > Components: Back End >Affects Versions: 2.2.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.2.0 > > > It is classifying some valid characters as "overlong" and erroring out. > The PNG schema on DFDLSchemas github has 1 test that runs into this bug on 3 > byte Devangari script characters. > This is 6 devangari characters: e0 a4 b6 e0 a5 80 e0 a4 b0 e0 a5 8d e0 a4 b7 > e0 a4 95 > Should be: शीर्षक > But is coming out all substitution chars. > In 3 byte utf-8, the bits that at least one of must be non-zero are shown > here in M, notice one of them is in the second byte. This second byte wasn't > being tested. > 1110 10Mx 10xx > In 4 byte utf-8, the bits that must at least one of be non-zero are: > 0 MMM 10MM 10xx 10xx -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1969) Exception on unparse of string with explicit length and truncateSpecifiedLengthString="yes"
[
https://issues.apache.org/jira/browse/DAFFODIL-1969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Thompson closed DAFFODIL-1969.
---
Pulled latest updates from incubator-daffodil repository which included
specified commit, 0872aa27743a30279353dc30523e7ee1b568cdfd.
Verified daffodil builds and executes all sbt tests successfully, including the
associated test case added in the specified commit.
Verified changes specified in the commit comment.
All nightly tests also executed successfully.
> Exception on unparse of string with explicit length and
> truncateSpecifiedLengthString="yes"
> ---
>
> Key: DAFFODIL-1969
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1969
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 2.1.0
> Environment: Windows 7, Daffodil 2.1.0
>Reporter: James L Welch
>Assignee: Steve Lawrence
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: AppTable.xml, TFconfigSYS.dfdl.xsl, appTable.cfg, u.bat
>
>
> java.lang.StringIndexOutOfBoundsException: String index out of range: 64
> dfdl:length="64" dfdl:textOutputMinLength="64" />
> Removed dfdl:textOutputMinLength - same error
> Attached file AppTable.cfg is the original binary file
> {color:#FF}parsed{color} into AppTable.xml
> Unparsing gives exception on first AppParms element
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1970) Exception instead of error message - expression variable on wrong level
[
https://issues.apache.org/jira/browse/DAFFODIL-1970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Thompson closed DAFFODIL-1970.
---
Closing in lieu of duplicate ticket DAFFODIL-1025 per dev.
> Exception instead of error message - expression variable on wrong level
> ---
>
> Key: DAFFODIL-1970
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1970
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 2.1.0
> Environment: Windows 7; dfdl: 2.1.0
>Reporter: James L Welch
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: TFconfigBUG.dfdl.xsl, error.txt, x.bat
>
>
> org.apache.daffodil.exceptions.Abort: Invariant broken:
> RelativePathExpression.this.steps2.apply(0).isInstanceOf[org.apache.daffodil.dpath.Up]
> DFDL file dfdl:occursCount="\{./Reg_Count}"
> should be dfdl:occursCount="\{../Reg_Count}"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1966) Excessive memory usage in CLI performance command
[ https://issues.apache.org/jira/browse/DAFFODIL-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1966. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 522a6d88e7b17ce200e4d8198dd7090eb6f10099. Verified daffodil builds and executes all sbt tests. Verified changes specified in the commit comment included. Verified all nightly tests now execute successfully without memory issues. > Excessive memory usage in CLI performance command > - > > Key: DAFFODIL-1966 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1966 > Project: Daffodil > Issue Type: Bug > Components: Performance >Affects Versions: 2.2.0 >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Critical > Fix For: 2.2.0 > > > The CLI performance command is using an excessive amount of memory, resulting > in out of memory exceptions. Needs to investigate and fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DAFFODIL-931) Variable-width charset with 'replace' can result in wrong length calculations
[
https://issues.apache.org/jira/browse/DAFFODIL-931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625732#comment-16625732
]
Dave Thompson commented on DAFFODIL-931:
Also ran manual parse/unparse streaming tests using VMF and PCAP formats.
> Variable-width charset with 'replace' can result in wrong length calculations
> -
>
> Key: DAFFODIL-931
> URL: https://issues.apache.org/jira/browse/DAFFODIL-931
> Project: Daffodil
> Issue Type: Bug
> Components: Back End, General
>Affects Versions: s12
>Reporter: Michael Beckerle
>Assignee: Steve Lawrence
>Priority: Major
> Fix For: 2.2.0
>
>
> Given a utf-8 string with a single-byte non-decodable byte in the middle.
> When we parse this the non-decodable byte will contribute a unicode
> replacement character to the string. 0xFFFD is the character code.
> If you then take this string and call getBytes("utf-8") on it, you will not
> get the right length. You will get 3 instead of 1 for the error because
> 0xFFFD takes 3 bytes in utf-8.
> The way we are measuring how far to move ahead in bytes right now, when we
> have a variable-width encoding like UTF-8, is to do exactly the above, call
> getBytes to find how long the string was.
> This will cause us to move too far ahead into the data.
> Test case to illustrate is TBD, but isn't too hard to put together. Just put
> a string per above with length coming from an expression. Put the string
> between two binary int fields. The binary int field after will not be parsed
> properly. because we will advance too far on the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DAFFODIL-1065) parser: API needs to enable repeated calls to parser - not treat unconsumed data as 'left over'
[ https://issues.apache.org/jira/browse/DAFFODIL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625726#comment-16625726 ] Dave Thompson commented on DAFFODIL-1065: - Also ran manual parse/unparse streaming tests using VMF and PCAP formats. > parser: API needs to enable repeated calls to parser - not treat unconsumed > data as 'left over' > --- > > Key: DAFFODIL-1065 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1065 > Project: Daffodil > Issue Type: Improvement > Components: API >Affects Versions: s14 >Reporter: Michael Beckerle >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.2.0 > > > From an Email thread with James Garriss: > While I got the workaround working, I definitely think this is an improvement > opportunity for your API. It should have simple methods that: > �� Detect a ���leftover data��� condition. > �� Return some sort of pointer to the location of the leftover data. > �� Return the entirety of the leftover data. > This would be super helpful for anyone trying to use Daffodil in a filter. > -Original Message- > From: Steve Lawrence [mailto:slawre...@tresys.com] > Sent: Wednesday, October 08, 2014 10:05 AM > ... > Subject: Re: Daffodil 0.14 - warning about extra, unprocessed data > Yep, it looks like our CLI has two ways we determine if there is left > over data. > The first is the isAtEnd method. The problem with this is that I think > it only works if you provide the size of your data to the parse() > method. Otherwise, it will always return false. This is a bug. However, > the workaround is to get the size of your input data, and pass it to the > parse function, e.g.: > ParseResult pr = processorFactory.parse(input, lengthOfInput) > The second way is to read the internal state to determine the length of > the data and compare that with the resulting data location. This is what > the Scala CLI does when the length isn't known. The problem with this > method is that it is internal information that is not available via the > Java API, so while it works for our CLI written in Scala, it can't be > used in your GUI. I'm not sure this information should be available via > the API, I think we just need to fix the first bug. > So the short of it, pass in the length of your data and I think isAtEnd > will work. > - Steve -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1065) parser: API needs to enable repeated calls to parser - not treat unconsumed data as 'left over'
[ https://issues.apache.org/jira/browse/DAFFODIL-1065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1065. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 07ee2434bb207d7137dbe4ea70efb41921562334. Verified daffodil builds and executes all sbt tests successfully, including the associated test case added in the specified commit. Verified changes specified in the commit comment. All nightly tests also executed successfully. > parser: API needs to enable repeated calls to parser - not treat unconsumed > data as 'left over' > --- > > Key: DAFFODIL-1065 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1065 > Project: Daffodil > Issue Type: Improvement > Components: API >Affects Versions: s14 >Reporter: Michael Beckerle >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.2.0 > > > From an Email thread with James Garriss: > While I got the workaround working, I definitely think this is an improvement > opportunity for your API. It should have simple methods that: > �� Detect a ���leftover data��� condition. > �� Return some sort of pointer to the location of the leftover data. > �� Return the entirety of the leftover data. > This would be super helpful for anyone trying to use Daffodil in a filter. > -Original Message- > From: Steve Lawrence [mailto:slawre...@tresys.com] > Sent: Wednesday, October 08, 2014 10:05 AM > ... > Subject: Re: Daffodil 0.14 - warning about extra, unprocessed data > Yep, it looks like our CLI has two ways we determine if there is left > over data. > The first is the isAtEnd method. The problem with this is that I think > it only works if you provide the size of your data to the parse() > method. Otherwise, it will always return false. This is a bug. However, > the workaround is to get the size of your input data, and pass it to the > parse function, e.g.: > ParseResult pr = processorFactory.parse(input, lengthOfInput) > The second way is to read the internal state to determine the length of > the data and compare that with the resulting data location. This is what > the Scala CLI does when the length isn't known. The problem with this > method is that it is internal information that is not available via the > Java API, so while it works for our CLI written in Scala, it can't be > used in your GUI. I'm not sure this information should be available via > the API, I think we just need to fix the first bug. > So the short of it, pass in the length of your data and I think isAtEnd > will work. > - Steve -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-931) Variable-width charset with 'replace' can result in wrong length calculations
[
https://issues.apache.org/jira/browse/DAFFODIL-931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Thompson closed DAFFODIL-931.
--
Pulled latest updates from incubator-daffodil repository which included
specified commit, 07ee2434bb207d7137dbe4ea70efb41921562334.
Verified daffodil builds and executes all sbt tests successfully, including the
associated test case added in the specified commit.
Verified changes specified in the commit comment.
All nightly tests also executed successfully.
> Variable-width charset with 'replace' can result in wrong length calculations
> -
>
> Key: DAFFODIL-931
> URL: https://issues.apache.org/jira/browse/DAFFODIL-931
> Project: Daffodil
> Issue Type: Bug
> Components: Back End, General
>Affects Versions: s12
>Reporter: Michael Beckerle
>Assignee: Steve Lawrence
>Priority: Major
> Fix For: 2.2.0
>
>
> Given a utf-8 string with a single-byte non-decodable byte in the middle.
> When we parse this the non-decodable byte will contribute a unicode
> replacement character to the string. 0xFFFD is the character code.
> If you then take this string and call getBytes("utf-8") on it, you will not
> get the right length. You will get 3 instead of 1 for the error because
> 0xFFFD takes 3 bytes in utf-8.
> The way we are measuring how far to move ahead in bytes right now, when we
> have a variable-width encoding like UTF-8, is to do exactly the above, call
> getBytes to find how long the string was.
> This will cause us to move too far ahead into the data.
> Test case to illustrate is TBD, but isn't too hard to put together. Just put
> a string per above with length coming from an expression. Put the string
> between two binary int fields. The binary int field after will not be parsed
> properly. because we will advance too far on the string.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1736) Very large BigInt unparses to zero.
[ https://issues.apache.org/jira/browse/DAFFODIL-1736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1736. --- Pulled latest updates to the incubator-daffodil repository which include commit 6b5fa10af9538924f3c811c40e94905b309980c1 (correct commit). Verified the affected tests execute successfully, now round tripping as expected. > Very large BigInt unparses to zero. > --- > > Key: DAFFODIL-1736 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1736 > Project: Daffodil > Issue Type: Bug > Components: Back End >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.2.0 > > > See test introduction_1_02. DFDL-1707 enabled support for unparsing BigInt's. > However, if you modify introduction_1_02 to roundTrip it fails. This appears > to be because it is trying to unparse 8.6E-200. It looks like when ICU > converts this to text, it results in a zero. Perhaps it's a limit of ICU, or > perhaps it's a bug. Should look into if updating to a newer version of ICU > fixes this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1962) TravisCI builds fail with error 137 out of memory
[ https://issues.apache.org/jira/browse/DAFFODIL-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1962. --- Pulled latest updates from incubator-daffodil repository which included specified commit, f5bace6f483297b8723ccd4327fd0caa22772dc3. Verified daffodil builds and executes all sbt tests successfully. Verified changes specified in the commit comment. All nightly tests also executed successfully. > TravisCI builds fail with error 137 out of memory > - > > Key: DAFFODIL-1962 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1962 > Project: Daffodil > Issue Type: Improvement > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.2.0 > > > The Travis CI builds seem to fail randomly with a "Killed" message and error > 137. It seems this is do to running out of memory. > We should investigate if we're used excessive amounts of memory during > testing, or determine if there is a way to bump the available memory given by > the TravisCI build system. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1652) Scala 2.12 Upgrade
[ https://issues.apache.org/jira/browse/DAFFODIL-1652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1652. --- Pulled latest updates from incubator-daffodil repository which included specified commit, f1e47b40779c6f6955d4faf580d6a63d0ae3d0d6. Verified daffodil builds and executes all sbt tests successfully, including the associated test case added in the specified commit. Verified changes specified in the commit comment. All nightly tests also executed successfully. > Scala 2.12 Upgrade > -- > > Key: DAFFODIL-1652 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1652 > Project: Daffodil > Issue Type: Improvement > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.2.0 > > Attachments: daffodil-2.2.0.patch > > > Scala 2.12 is out. This comes with various new features, as well as improved > macros. We should investigate if this is worth the upgrade and test it out. > Note that 2.12 requires Java 8. We require Java 8 due to some encoding bugs, > but we could theoretically support Java7 by detected Java version and taking > different code paths where the bug exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1937) Make the Apache RAT check more automated
[ https://issues.apache.org/jira/browse/DAFFODIL-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1937. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 1ec6abb3c74279f2eaa0204568458a459cdb4ef6. Verified daffodil builds and executes all sbt tests successfully, including the associated test case added in the specified commit. Verified changes specified in the commit comment. All nightly tests also executed successfully. > Make the Apache RAT check more automated > > > Key: DAFFODIL-1937 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1937 > Project: Daffodil > Issue Type: Improvement > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Critical > Fix For: 2.2.0 > > > Running RAT on Daffodil is currently a manual process that reqiures > downloading an Apache RAT release and using the right command line arguments. > This requires too much effort and so contributors will forget to run it, and > we cannot run checks on the automated build. > An sbt-rat plugin started development on github > (https://github.com/jvz/sbt-rat). We should investigate if this meets our > needs and incorporate it into Daffodil. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1738) Implement textNumberRep='zoned'
[ https://issues.apache.org/jira/browse/DAFFODIL-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1738. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 098b3d54e6bc6d60ffbb612e912ee2daec83dcca. Verified daffodil builds and executes all sbt tests successfully, including the associated test case added in the specified commit. All nightly tests also executed successfully. > Implement textNumberRep='zoned' > --- > > Key: DAFFODIL-1738 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1738 > Project: Daffodil > Issue Type: New Feature > Components: DFDL Language >Reporter: Elizabeth Fahl >Assignee: Dave Thompson >Priority: Major > Fix For: 2.2.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-99) Date, Time, DateTime - Binary - binaryCalendarRep='binary'
[ https://issues.apache.org/jira/browse/DAFFODIL-99?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-99. - Pulled latest updates from incubator-daffodil repository which included specified commit, b3c184995d251021edd65d559bd3726d553c1f8f. Verified daffodil builds and executes all sbt tests successfully, including the associated test case added in the specified commit. All nightly tests also executed successfully. > Date, Time, DateTime - Binary - binaryCalendarRep='binary' > -- > > Key: DAFFODIL-99 > URL: https://issues.apache.org/jira/browse/DAFFODIL-99 > Project: Daffodil > Issue Type: New Feature > Components: DFDL Language, Front End >Reporter: Jeremy Solt >Assignee: Dave Thompson >Priority: Major > Fix For: 2.2.0 > > > Also involved. Property binaryCalendarEpoch. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1924) DaffodilXMLLoader needs to explicitly specify which instance to use
[ https://issues.apache.org/jira/browse/DAFFODIL-1924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1924. --- Pulled latest updates from incubator-daffodil repository which included specified commit, e79a43c8ce8a19992811605cf1b4f47042e071d5. Verified daffodil builds and executes all sbt tests successfully, including the associated test case added in the specified commit. > DaffodilXMLLoader needs to explicitly specify which instance to use > --- > > Key: DAFFODIL-1924 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1924 > Project: Daffodil > Issue Type: New Feature > Components: Front End >Affects Versions: 2.1.0 >Reporter: Steve Lawrence >Assignee: Dave Thompson >Priority: Major > Fix For: 2.2.0 > > > The SAXParserFactory.newInstance() method is used to create a new instance of > a SAXParser. There are a handful of steps taken to find a SAXParser to use, > as documented in the [newInstance() > JavaDocs|https://docs.oracle.com/javase/7/docs/api/javax/xml/parsers/SAXParserFactory.html#newInstance()]. > The problem with this is that Daffodil sets the entity-resolver property to > an instance of the DFDLCatalogResolver, which extends a specific Xerces > XMLEntityResolver. In order for this to work, we must be guaranteed that the > SAXParserFactory instance we get is the Xerces one, and not some other one > overridden by one of the ways described in newInstace(). > So instead, we should use the newInstance method that lets you provide a > string specifying which factory class to use, and specify our Xerces one. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1918) Update JSON schema in tests to use Boolean type
[ https://issues.apache.org/jira/browse/DAFFODIL-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1918. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 2a4c81ac76ee6abd49ffa8bff40d432e63a1f5b6. Per comment in ticket the JSON schema has been removed. Verified that the JSON schema and test files have been removed and daffodil successfully builds and executes all sbt test cases. > Update JSON schema in tests to use Boolean type > --- > > Key: DAFFODIL-1918 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1918 > Project: Daffodil > Issue Type: Improvement > Components: Clean Ups, DFDL Schemas, QA >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.2.0 > > > The JSON schema has a boolean hack in it. This should be replaced by use of > real boolean type now that it is fully implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1805) Support AIS format ascii armoring and 6-bit encoding
[ https://issues.apache.org/jira/browse/DAFFODIL-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1805. --- Pulled latest updates from incubator-daffodil repository which included specified commit, a705789fd0c37963371e188350e53279eac1e360. Verified daffodil successfully builds and executes all sbt tests, including the related test cases. > Support AIS format ascii armoring and 6-bit encoding > > > Key: DAFFODIL-1805 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1805 > Project: Daffodil > Issue Type: New Feature > Components: Back End, DFDL Language >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.2.0 > > > AIS = Automated Identification System, a marine system in heavy use. > aka the ITU M.1371-2 Standard. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1734) base64 encoding and other preprocessing
[ https://issues.apache.org/jira/browse/DAFFODIL-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1734. --- Pulled latest updates from incubator-daffodil repository which included specified commit, 1ea2290f28e9ede344cb7d8d06f38a7fe818dfc3. Verified daffodil successfully builds and executes all sbt tests, including the related test cases. > base64 encoding and other preprocessing > --- > > Key: DAFFODIL-1734 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1734 > Project: Daffodil > Issue Type: New Feature > Components: DFDL Language >Reporter: Michael Beckerle >Assignee: Michael Beckerle >Priority: Major > Fix For: 2.2.0 > > > Many formats have payloads that are embedded base64-encoded data. > We need a DFDL language feature enabling us to specify that the underlying > bytes of an element are base64 encoded. That is, the data stream is base64 > encoded, and must be decoded and then parsed, or unparsed and then encoded > into base64 text. > A similar preprocessing should handle compression, as formats like VMF > (mil-std-6017) can have compressed payloads. > In general, we should enable these pre-processing encoder/decoders to be > pluggable so the set can be extended easily. > This is a form of layering. If we consider just parsing for a moment, this > enables one to define an element which is a byte-array or string, and then > treat that as the input source for subsequent parsing without having to > explicitly do a second pass of calling daffodil to parse the element. You can > get Daffodil to do this second pass itself automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1430) macroLib should not be needed after scala compilation, but seems to be required
[ https://issues.apache.org/jira/browse/DAFFODIL-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1430. --- Concur, verified as part of JIRA ticket DAFFODIL-1857 verification. Closing this ticket. > macroLib should not be needed after scala compilation, but seems to be > required > --- > > Key: DAFFODIL-1430 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1430 > Project: Daffodil > Issue Type: Improvement > Components: Infrastructure >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.2.0 > > > We now have this new daffodil-macro-lib module. > In the projects/build.scala, it is *not* listed as "nopub". It seems to be > needed to be published, but I don't know why it would be needed. > In creating a scala build for a data format (where the jar created contains > the DFDL schema, and can run TDML tests), there is a dependency on the > daffodil-tdml library, and this transitively ends up requiring the macroLib, > so fails if the projects/build.scala is changed so that macroLib is "nopub". > So the task here is figure out why, so that macroLib truly is only needed at > scala-compile time for daffodil, and not needed when other consumers of > daffodil libraries are used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1714) ULong modulus operator broken
[
https://issues.apache.org/jira/browse/DAFFODIL-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Thompson closed DAFFODIL-1714.
---
Pulled latest updates from incubator-daffodil repository which included
specified commit, 98874596fdf83ad272053c62af83a0a6b71df3e2.
Verified daffodil successfully builds and executes all sbt tests, including the
new related tests.
> ULong modulus operator broken
> -
>
> Key: DAFFODIL-1714
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1714
> Project: Daffodil
> Issue Type: Bug
> Components: General, Libraries
>Affects Versions: 2.0.0
>Reporter: Steve Lawrence
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.2.0
>
>
> It appears that the ULong modulus operator is broken. If we run this code:
> {code}
> for (i <- 0 to 16 ) {
> val numerator = ULong(i)
> val denominator = ULong(8)
> val remainder = numerator % denominator
> println(i + " % 8 = " + remainder)
> }
> {code}
> we get the result:
> {code}
> 0 % 8 = 0
> 1 % 8 = 1
> 2 % 8 = 2
> 3 % 8 = 3
> 4 % 8 = 4
> 5 % 8 = 5
> 6 % 8 = 6
> 7 % 8 = 7
> 8 % 8 = 8
> 9 % 8 = 9
> 10 % 8 = 10
> 11 % 8 = 11
> 12 % 8 = 12
> 13 % 8 = 13
> 14 % 8 = 14
> 15 % 8 = 15
> 16 % 8 = 0
> {code}
> Anything % 8 should always be less than 8. But some results are greater than
> 8. Not sure exactly what is going on, and the generated bytecode is really
> hard to follow. In fact, I wouldn't be surprised if ULongs are actually
> slowing things down. The byte code was kindof a mess. We might want to
> revisit ULongs and how else to support > 4GB files.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1933) Performance regression with new Base64/Layering changes
[
https://issues.apache.org/jira/browse/DAFFODIL-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Thompson closed DAFFODIL-1933.
---
Pulled latest updates from incubator-daffodil repository which included
specified commit, 66b20173a1a92fcfd0688f87b4cb01d2e5adff84.
Verified that the performance improved to previous values +- normal deviation.
> Performance regression with new Base64/Layering changes
> ---
>
> Key: DAFFODIL-1933
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1933
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Reporter: Steve Lawrence
>Assignee: Dave Thompson
>Priority: Critical
> Fix For: 2.2.0
>
>
> Nightly tests showed a fairly drastic performance regression (50% reduction
> in speed) in commit 1ea2290f28: All properties for Base64/layering
> implemented.
> Skimming the code, one change jumped out at me as a likely cause in
> ByteBufferDataInputStream.scala:
> {code:language=diff}
> @@ -91,7 +90,13 @@ object ByteBufferDataInputStream {
>case _ => {
> // copy the contents of the stream into an array of bytes
> val bos = new ByteArrayOutputStream
> -IOUtils.copy(in, bos)
> +var b: Int = 0
> +while ({
> + b = in.read()
> + b != -1
> +}) {
> + bos.write(b)
> +}
> bos.flush()
> bos.close()
> in.close()
> {code}
> This changes the copy from a bulk copy to a byte-by-byte copy, which appears
> to have pretty drastic performance. A few simple quick tests shows that
> reverting this change recovers all performance losses.
> Mike, can you confirm if this change can be reverted, or if there was some
> other reason related to layering to switch to a byte-by-byte copy?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1925) Remove testWSPStar.dfdl.xsd and json5.dfdl.xsd
[ https://issues.apache.org/jira/browse/DAFFODIL-1925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1925. --- Pulled latest updates from incubator-daffodil repository which included the specified commit, 2a4c81ac76ee6abd49ffa8bff40d432e63a1f5b6. Verified that the specified files were no longer included and daffodil successfully builds and executes all sbt tests. > Remove testWSPStar.dfdl.xsd and json5.dfdl.xsd > -- > > Key: DAFFODIL-1925 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1925 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Affects Versions: 2.1.0 >Reporter: Steve Lawrence >Assignee: Dave Thompson >Priority: Major > Fix For: 2.2.0 > > > These two files were contributions from Mitre. We never attempted to get an > SGA from Mitre since we thought all contributions were removed/replaced. This > means that these files were improperly relicensed to Apache License v2. The > tests do not provide much value, so should just be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1936) Prepare for Daffodil 2.2.0 development
[ https://issues.apache.org/jira/browse/DAFFODIL-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1936. --- Pulled latest updates from incubator-daffodil repository which included specified commit, dbe98c1065d6874471f271124627fbe3e346976f. Verified that the has been updated to 2.2.0. > Prepare for Daffodil 2.2.0 development > -- > > Key: DAFFODIL-1936 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1936 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Dave Thompson >Priority: Major > Fix For: 2.2.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1921) option to bypass Java 8 check
[ https://issues.apache.org/jira/browse/DAFFODIL-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1921. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included the specified commit, 3a5183a40a3df65219f68478fc8a662eab9cf3f4. Verified by review of source files associated with this update. > option to bypass Java 8 check > - > > Key: DAFFODIL-1921 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1921 > Project: Daffodil > Issue Type: Improvement > Components: Back End >Affects Versions: 2.1.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > Currently Daffodil insists on Java 8+. It refuses to run on Java 7. > Some users need it to run on Java 7, and despite the possible charset decoder > issues on decode errors in Java 7, they want to try it anyway. > So some mechanism for bypassing the Java 8 check should be put in place so > that the check will not be done. I would suggest an undocumented "feature" > for now, as we don't really know the implications of running on Java 7, and > don't test there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1920) toString representation of grammar prims and parser/unparsers need improvement
[ https://issues.apache.org/jira/browse/DAFFODIL-1920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1920. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included the specified commit, 8a29c60b5c5857f17bc63298345fbde24a6d0a8c. Verified by review of source files associated with this update. > toString representation of grammar prims and parser/unparsers need improvement > -- > > Key: DAFFODIL-1920 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1920 > Project: Daffodil > Issue Type: Improvement > Components: Clean Ups >Affects Versions: 2.1.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > It's hard to debug because the printed quasi-XML that we use for the grammar > objects and for the parser/unparser objects is not uniform enough. > Improve these so that it's easier to see what is going on when debugging, or > looking at a trace. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1838) Need more warning suppression options
[ https://issues.apache.org/jira/browse/DAFFODIL-1838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1838. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included the specified commit, 075084f5a825bde3168b8e89a02c7e58404cc325. Verified that all sbt tests execute successfully. Verified that all nightly tests execute successfully. Verified that noEmptyDefault warnings can now be suppressed. > Need more warning suppression options > - > > Key: DAFFODIL-1838 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1838 > Project: Daffodil > Issue Type: Bug >Reporter: Steve Lawrence >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > We currently have two warnings that can specifically be suppress: multiple > choice branches and escape scheme ref undefined. We have many others that > cannot be ignore without the use of "all". We should enhance of warnings so > that they require a warn id so all similar warnings can be suppressed without > resorting to disabling all warnings. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1874) Incorrect warning message about default value and no empty representation
[
https://issues.apache.org/jira/browse/DAFFODIL-1874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Thompson closed DAFFODIL-1874.
---
Pulled the latest updates from the incubator-daffodil repository and verified
the updates included the specified commit,
075084f5a825bde3168b8e89a02c7e58404cc325.
Verified that the warning messages displyed during NACHA format processing now
reflect the specified change. Verified that all sbt tests execute successfully.
Verified that all nightly tests execute successfully.
> Incorrect warning message about default value and no empty representation
> -
>
> Key: DAFFODIL-1874
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1874
> Project: Daffodil
> Issue Type: Bug
> Components: Middle End
>Affects Versions: 2.0.0
>Reporter: Michael Beckerle
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.1.0
>
>
> NACHA schema (on DFDLSchemas github) has a ticket about suppressing a warning
> that we get for almost every element in that schema:
> https://github.com/DFDLSchemas/NACHA/issues/2
> This is the warning we get:
> {{{Schema Definition Warning: Element has no empty representation so cannot
> have XSD default='0' as a default value.
> Schema context: element reference {ach:2013}PriorityCode Location line 55
> column 18 in
> file:/home/mbeckerle-unencrypted/DFDLSchemas/NACHA/2013/nacha_records.xsd}}}
> The warning is misleading, it should say that for an element with no empty
> rep the default value can only be applied on unparsing (where the criteria is
> that the element is not in the infoset).
> AND that corrected warning, which we'll still get for every element of NACHA,
> needs to be suppressable for a schema like NACHA where otherwise you get
> dozens of them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1473) DFDL Schema Validation not happening properly
[ https://issues.apache.org/jira/browse/DAFFODIL-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1473. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included the specified commit, 6ad629d19b9c53db117155ed4f99ec565a07e274. Verified by review that that there is now a call to .setValidation in DFDLSchemaFile.scala. Verified that all tests associated with this commit execute successfully. > DFDL Schema Validation not happening properly > - > > Key: DAFFODIL-1473 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1473 > Project: Daffodil > Issue Type: Bug > Components: Front End, General >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > I discovered that the natural xerces load time validation that is supposed to > happen when DFDL schemas are loaded, is not happening. > There seems to be code for doing a whole separate load pass just to validate > rather than doing it at load time. This might be a holdover from when we were > using the "constructing parser" for XML for more things, rather than Xerces. > Not sure. > All I know is that in DFDLSchemaFile.scala, there is no call to > loader.setValidation(...) so those loads don't validate, and that causes many > syntactic problems to go undetected. > However, I don't want to fix this right now, because lots of tests break. We > should fix this one little thing and fix up the tests that break at that > point in one commit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DAFFODIL-1893) TDML Runner not caching schema compiles - only caches schemas if compileAllTopLevel is true
[ https://issues.apache.org/jira/browse/DAFFODIL-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16431093#comment-16431093 ] Dave Thompson commented on DAFFODIL-1893: - Pulled the latest updates from the incubator-daffodil repository and verified the updates included the specified commit, 836d657ca5bd99577110e8c30007591a26ee4317. Talked with Steve L about how to verify and he said that he had checked the performance during previous review and verified that performance for tests with long schema compile has improved. > TDML Runner not caching schema compiles - only caches schemas if > compileAllTopLevel is true > --- > > Key: DAFFODIL-1893 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1893 > Project: Daffodil > Issue Type: Bug > Components: TDML Runner >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > The TDML runner is supposed to cache compiled schemas so that many tests > using the same DFDL schema will compile it only once. > It currently only does this when compileAllTopLevel is true, but the default > for this is false. > The cache should be modified to take the root element into account - i.e., > compile all top levels need not be true. The cache should hit, avoiding > recompilation, so long as the same root element is being used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1893) TDML Runner not caching schema compiles - only caches schemas if compileAllTopLevel is true
[ https://issues.apache.org/jira/browse/DAFFODIL-1893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1893. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included the specified commit, 836d657ca5bd99577110e8c30007591a26ee4317. Verified that run time for tests with long schema compile times and use a root element has improved. > TDML Runner not caching schema compiles - only caches schemas if > compileAllTopLevel is true > --- > > Key: DAFFODIL-1893 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1893 > Project: Daffodil > Issue Type: Bug > Components: TDML Runner >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > The TDML runner is supposed to cache compiled schemas so that many tests > using the same DFDL schema will compile it only once. > It currently only does this when compileAllTopLevel is true, but the default > for this is false. > The cache should be modified to take the root element into account - i.e., > compile all top levels need not be true. The cache should hit, avoiding > recompilation, so long as the same root element is being used. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DAFFODIL-1912) Daffodil commit 6ad629d19b9c53db117155ed4f99ec565a07e274 causes the gmtif schema fail to compile.
Dave Thompson created DAFFODIL-1912:
---
Summary: Daffodil commit 6ad629d19b9c53db117155ed4f99ec565a07e274
causes the gmtif schema fail to compile.
Key: DAFFODIL-1912
URL: https://issues.apache.org/jira/browse/DAFFODIL-1912
Project: Daffodil
Issue Type: Bug
Components: General
Affects Versions: 2.2.0
Reporter: Dave Thompson
{color:#00}Pulled the latest updates to incubatlr-daffodil repository. The
gmtif schema failed to compile during the nightly test run.{color}
{color:#00}{color:#00}Rolled back to commit
075084f5a825bde3168b8e89a02c7e58404cc325 before the gmtif schema/parser
successfully compiled.{color}{color}
{color:#00}{color:#00}Daffodil commit
6ad629d19b9c53db117155ed4f99ec565a07e274 causes the gmtif schema to fail to
compile.{color}{color}
{color:#00}{color:#00}Numerous schema validation errors are returned. A
subset shown below. {color}{color}
{color:#00}*Error messages are below:* {color}
{color:#00}Attempting to save parser for gmtif.{color}
{color:#00}Test name is: gmtif_1t_76b_2{color}
{color:#00}CMD:
/home/dfdl/incubator-daffodil/daffodil-cli/target/universal/stage/bin/daffodil
-v save-parser -s
/home/dfdl/dfdl-dataformats/data-formats/gmtif/src/main/resources/com/tresys/gmtif/xsd/gmtif.dfdl.xsd
> /home/dfdl/dfdl-testharness/saved_parsers/0.0.0/gmtif.parser.bin{color}
{color:#00}process return code: 1{color}
{color:#00}Failed to Save Parser - gmtif{color}
{color:#00} {color}
{color:#00}Error: [error] Schema Definition Error: Error loading schema due
to org.xml.sax.SAXParseException; systemId:
file:/home/dfdl/dfdl-dataformats/data-formats/gmtif/src/main/resources/com/tresys/gmtif/xsd/gmtif_dwell_segment.dfdl.xsd;
lineNumber: 220; columnNumber: 56; src-resolve: Cannot resolve the name
'gmtif:I16' to a(n) 'type definition' component.{color}
{color:#00}Schema context:
file:/home/dfdl/dfdl-dataformats/data-formats/gmtif/src/main/resources/com/tresys/gmtif/xsd/gmtif_dwell_segment.dfdl.xsd
Location in file:??{color}
{color:#00}[error] Schema Definition Error: Error loading schema due to
org.xml.sax.SAXParseException; systemId:
file:/home/dfdl/dfdl-dataformats/data-formats/gmtif/src/main/resources/com/tresys/gmtif/xsd/gmtif_dwell_segment.dfdl.xsd;
lineNumber: 230; columnNumber: 54; src-resolve: Cannot resolve the name
'gmtif:I16' to a(n) 'type definition' component.{color}
{color:#00}Schema context:
file:/home/dfdl/dfdl-dataformats/data-formats/gmtif/src/main/resources/com/tresys/gmtif/xsd/gmtif_dwell_segment.dfdl.xsd
Location in file:??{color}
{color:#00}[error] Schema Definition Error: Error loading schema due to
org.xml.sax.SAXParseException; systemId:
file:/home/dfdl/dfdl-dataformats/data-formats/gmtif/src/main/resources/com/tresys/gmtif/xsd/gmtif_dwell_segment.dfdl.xsd;
lineNumber: 259; columnNumber: 61; src-resolve: Cannot resolve the name
'gmtif:I16' to a(n) 'type definition' component.{color}
{color:#00}Schema context:
file:/home/dfdl/dfdl-dataformats/data-formats/gmtif/src/main/resources/com/tresys/gmtif/xsd/gmtif_dwell_segment.dfdl.xsd
Location in file:??{color}
{color:#00}.{color}
{color:#00}.{color}
{color:#00}.{color}
{color:#00}.{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (DAFFODIL-1908) Three daffodil-test sbt tests intermittently fail
Dave Thompson created DAFFODIL-1908: --- Summary: Three daffodil-test sbt tests intermittently fail Key: DAFFODIL-1908 URL: https://issues.apache.org/jira/browse/DAFFODIL-1908 Project: Daffodil Issue Type: Bug Components: TDML Runner Affects Versions: 2.1.0 Reporter: Dave Thompson Pulled latest updates from incubator-daffodil repository, commit 62eea54b75392c3fffcb36060d5bb078dfed3910. Thee 3 test are intermittently failing on the test machine. The tests passed 7 times and failed 4 (2 fail, 2 pass, 2 fail then 5 pass). Did a git clean after 1^st^ 2 fails. These tests have not failed on my VM. Tests and failure are below. [debug] Test org.apache.daffodil.section06.namespaces.TestNamespaces.test_multi_encoding_01 started [error] Test org.apache.daffodil.section06.namespaces.TestNamespaces.test_multi_encoding_01 failed: org.apache.daffodil.tdml.TDMLException: Schema Definition Error: Error loading schema due to org.xml.sax.SAXParseException; systemId: file:/home/dfdl/incubator-daffodil/daffodil-test/target/scala-2.11/test-classes/org/apache/daffodil/section06/namespaces/multi_base_09.dfdl.xsd; lineNumber: 1; columnNumber: 1; < expected [error] Schema context: file:/home/dfdl/incubator-daffodil/daffodil-test/target/scala-2.11/test-classes/org/apache/daffodil/section06/namespaces/multi_base_09.dfdl.xsd Location in file:?? [error] Schema Definition Error: No XML Node could be loaded from URISchemaSource(file:/home/dfdl/incubator-daffodil/daffodil-test/target/scala-2.11/test-classes/org/apache/daffodil/section06/namespaces/multi_base_09.dfdl.xsd). [error] Schema context: file:/home/dfdl/incubator-daffodil/daffodil-test/target/scala-2.11/test-classes/org/apache/daffodil/section06/namespaces/multi_base_09.dfdl.xsd Location in file:??, took 0.51 sec [error] at org.apache.daffodil.tdml.ParserTestCase$$anonfun$runProcessor$2.apply(TDMLRunner.scala:663) [error] at org.apache.daffodil.tdml.ParserTestCase$$anonfun$runProcessor$2.apply(TDMLRunner.scala:663) [error] at scala.util.Either$LeftProjection.foreach(Either.scala:302) [error] at org.apache.daffodil.tdml.ParserTestCase.runProcessor(TDMLRunner.scala:663) [error] at org.apache.daffodil.tdml.TestCase.run(TDMLRunner.scala:612) [error] at org.apache.daffodil.tdml.DFDLTestSuite.runOneTestWithDataVolumes(TDMLRunner.scala:325) [error] at org.apache.daffodil.tdml.DFDLTestSuite.runOneTest(TDMLRunner.scala:313) [error] at org.apache.daffodil.tdml.Runner.runOneTest(RunnerFactory.scala:107) [error] at org.apache.daffodil.section06.namespaces.TestNamespaces.test_multi_encoding_01(TestNamespaces.scala:150) [error] ... [error] Caused by: Schema Definition Error: Error loading schema due to org.xml.sax.SAXParseException; systemId: file:/home/dfdl/incubator-daffodil/daffodil-test/target/scala-2.11/test-classes/org/apache/daffodil/section06/namespaces/multi_base_09.dfdl.xsd; lineNumber: 1; columnNumber: 1; < expected [error] Schema context: file:/home/dfdl/incubator-daffodil/daffodil-test/target/scala-2.11/test-classes/org/apache/daffodil/section06/namespaces/multi_base_09.dfdl.xsd Location in file:?? [debug] Test org.apache.daffodil.section06.namespaces.TestNamespaces.test_multi_encoding_01 finished, took 0.512 sec Also these tests: [debug] Test org.apache.daffodil.section06.namespaces.TestNamespaces.test_multi_encoding_02 started [error] Test org.apache.daffodil.section06.namespaces.TestNamespaces.test_multi_encoding_02 failed: [debug] Test org.apache.daffodil.section06.namespaces.TestNamespaces.test_multi_encoding_03 started [error] Test org.apache.daffodil.section06.namespaces.TestNamespaces.test_multi_encoding_03 failed: -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1902) Daffodil disallows attributes on xs:appinfo element
[ https://issues.apache.org/jira/browse/DAFFODIL-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1902. --- Closing per dev's determination. > Daffodil disallows attributes on xs:appinfo element > --- > > Key: DAFFODIL-1902 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1902 > Project: Daffodil > Issue Type: Bug > Components: Front End >Affects Versions: 2.1.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > Users at NATO report that Daffodil doesn't tolerate the xs:appinfo element > carrying additional attributes. They want to combine appinfo DFDL attributes > with other appinfo for other purposes on the same schema. > (I will get a clarifying example.) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1901) Apache Rat cannot detect licenses for Passera, Scala libs, and W3C files
[ https://issues.apache.org/jira/browse/DAFFODIL-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1901. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included the specified commit, 00aa364bd7af29fd5ddfd8c8119fc4af9b925171. Verified the specified changes were implemented. > Apache Rat cannot detect licenses for Passera, Scala libs, and W3C files > > > Key: DAFFODIL-1901 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1901 > Project: Daffodil > Issue Type: Bug >Reporter: Steve Lawrence >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > This files are currently excluded in .rat-excludes because Apache Rat says > these files are not allow. It appears that it is not detecting the licenses > for these files. Need to figure out what Apache Rat is looking for and why it > is not detecting the licenses -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1899) Fix NOTICE and LICENSE files
[ https://issues.apache.org/jira/browse/DAFFODIL-1899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1899. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included the specified commit, 00aa364bd7af29fd5ddfd8c8119fc4af9b925171. Verified the specified changes were implemented. > Fix NOTICE and LICENSE files > > > Key: DAFFODIL-1899 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1899 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > The copyright portions that are in the LICENSE on the various licenses should > be moved to the NOTICE. The LICENSE should still include which files are > under the other licenses. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1897) Some Schemas fail to compile with latest updates (v2.1.0-rc1)
[ https://issues.apache.org/jira/browse/DAFFODIL-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1897. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included the specified commit, abf9ea5e0fa0cece332877a38918324229f82928. Verified that all schemas still compile and parser files are generated. Verified the nightly test suite still executes successfully. > Some Schemas fail to compile with latest updates (v2.1.0-rc1) > - > > Key: DAFFODIL-1897 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1897 > Project: Daffodil > Issue Type: Bug > Components: DFDL Schemas >Affects Versions: 2.1.0 >Reporter: Dave Thompson >Assignee: Dave Thompson >Priority: Blocker > Fix For: 2.1.0 > > > Attempted to execute the nightly test suite on incubator-daffodil commit > add8c5a6f82f25073475a4d6c7ae94fb5ca6e01a. > Four dfdl schemas failed to compile for the following formats: army-drrs, > ato-rep, csvMixedNarrow and pcap. > The command and error are provided below. > Attempting to save parser for army-drrs. > Test name is: army-drrs_10t_all_1000 > CMD: > /home/dfdl/incubator-daffodil/daffodil-cli/target/universal/stage/bin/daffodil > -v save-parser -s > /home/dfdl/ngf-dfdl/daffodil-perf/src/test/resources/edu/illinois/ncsa/daffodil/fouo_disa/army_drrs_lh/army_drrs_lh.dfdl.xsd > > /home/dfdl/dfdl-testharness/saved_parsers/0.0.0/army-drrs.parser.bin > process return code: 1 > Failed to Save Parser - army-drrs > Error: [error] Schema Definition Error: No schema document at location > file:/home/dfdl/ngf-dfdl/daffodil-perf/src/test/resources/org/apache/daffodil/fouo_disa/army_drrs_lh/army_drrs_lh.dfdl.xsd. > Schema context: Import Location in file:unknown > [warning] Schema Definition Warning: schemaLocation property uses deprecated > edu/illinois/ncsa/daffodil path instead of org/apache/daffodil. Converting to > new path. > Schema context: Import Location in file:unknown > [info] Time (compiling): 219ms > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1903) Eclipse classpath broken for daffodil-test and other tests - now must have daffodil-lib-unittest
[ https://issues.apache.org/jira/browse/DAFFODIL-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1903. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included the specified commit, f33208a3917f3ca35c5f2d9084ef175bc6af133d. Verified by review daffodil-lib-unittest has been added the applicable project classpath. > Eclipse classpath broken for daffodil-test and other tests - now must have > daffodil-lib-unittest > > > Key: DAFFODIL-1903 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1903 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Affects Versions: 2.1.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > Moving built-in-formats from daffodil-lib to daffodil-lib-unittest breaks > many things. Every eclipse project that depended on that file is broken and > must have its eclipse classpath modified. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1896) Modify artifact names to include "apache" and "incubating"
[ https://issues.apache.org/jira/browse/DAFFODIL-1896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1896. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included the specified commit, a56215bcd697706bae5ab14661d37019f2e67ab8. Verified that the specified changes were implemented. > Modify artifact names to include "apache" and "incubating" > -- > > Key: DAFFODIL-1896 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1896 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > For each, the release artifacts should be of the form > "apache-daffodil-2.1.0-incubating" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1881) Prepare 2.1.0 release
[ https://issues.apache.org/jira/browse/DAFFODIL-1881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1881. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included changes from the specified commit, add8c5a6f82f25073475a4d6c7ae94fb5ca6e01a. Verified changes specified in commit comment have been implemented. > Prepare 2.1.0 release > - > > Key: DAFFODIL-1881 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1881 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1890) Update user related links from the wiki to daffodil.apache.org
[ https://issues.apache.org/jira/browse/DAFFODIL-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1890. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included the specified commit, 044ce2898b82848f3e33232d6c9eb54ac52fedca. Verified that the links identified as non-functional now work.. > Update user related links from the wiki to daffodil.apache.org > -- > > Key: DAFFODIL-1890 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1890 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > User related documentation has been moved from the wiki to > daffodil.apache.org. Those pages should be removed from the wiki, so links > should be updated to reference the daffodil.apache.org pages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1897) Some Schemas fail to compile with latest updates (v2.1.0-rc1)
[ https://issues.apache.org/jira/browse/DAFFODIL-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1897. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included the specified commit, 7fac84af9e9c27f3a9959bb17f86c9f05f18d728. Verified that all schemas now compile and parser files are generated. Verified the nightly test suite executes successfully. > Some Schemas fail to compile with latest updates (v2.1.0-rc1) > - > > Key: DAFFODIL-1897 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1897 > Project: Daffodil > Issue Type: Bug > Components: DFDL Schemas >Affects Versions: 2.1.0 >Reporter: Dave Thompson >Assignee: Dave Thompson >Priority: Blocker > Fix For: 2.1.0 > > > Attempted to execute the nightly test suite on incubator-daffodil commit > add8c5a6f82f25073475a4d6c7ae94fb5ca6e01a. > Four dfdl schemas failed to compile for the following formats: army-drrs, > ato-rep, csvMixedNarrow and pcap. > The command and error are provided below. > Attempting to save parser for army-drrs. > Test name is: army-drrs_10t_all_1000 > CMD: > /home/dfdl/incubator-daffodil/daffodil-cli/target/universal/stage/bin/daffodil > -v save-parser -s > /home/dfdl/ngf-dfdl/daffodil-perf/src/test/resources/edu/illinois/ncsa/daffodil/fouo_disa/army_drrs_lh/army_drrs_lh.dfdl.xsd > > /home/dfdl/dfdl-testharness/saved_parsers/0.0.0/army-drrs.parser.bin > process return code: 1 > Failed to Save Parser - army-drrs > Error: [error] Schema Definition Error: No schema document at location > file:/home/dfdl/ngf-dfdl/daffodil-perf/src/test/resources/org/apache/daffodil/fouo_disa/army_drrs_lh/army_drrs_lh.dfdl.xsd. > Schema context: Import Location in file:unknown > [warning] Schema Definition Warning: schemaLocation property uses deprecated > edu/illinois/ncsa/daffodil path instead of org/apache/daffodil. Converting to > new path. > Schema context: Import Location in file:unknown > [info] Time (compiling): 219ms > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1859) Switch to apache.org namespace and Apache v2 license
[ https://issues.apache.org/jira/browse/DAFFODIL-1859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1859. --- Pulled the latest updates from the incubator-daffodil repository and verified the updates included changes from the specified commits. Verified all sbt test execute successfully. Verified nightly test suite executed successfully. > Switch to apache.org namespace and Apache v2 license > > > Key: DAFFODIL-1859 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1859 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > This includes changing the directory layout and package names to org.apache, > updating license files where appropriate and checking with Apache Rat. > This is put on hold until all SGA's are submitted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (DAFFODIL-1890) Update user related links from the wiki to daffodil.apache.org
[ https://issues.apache.org/jira/browse/DAFFODIL-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson reassigned DAFFODIL-1890: --- Assignee: Steve Lawrence (was: Dave Thompson) > Update user related links from the wiki to daffodil.apache.org > -- > > Key: DAFFODIL-1890 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1890 > Project: Daffodil > Issue Type: Bug >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.1.0 > > > User related documentation has been moved from the wiki to > daffodil.apache.org. Those pages should be removed from the wiki, so links > should be updated to reference the daffodil.apache.org pages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DAFFODIL-1868) TDML Runner defaultConfig cannot be embedded. Not checked properly.
[ https://issues.apache.org/jira/browse/DAFFODIL-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358870#comment-16358870 ] Dave Thompson commented on DAFFODIL-1868: - Correction: Latest updates were add8c5a6f82f25073475a4d6c7ae94fb5ca6e01a. > TDML Runner defaultConfig cannot be embedded. Not checked properly. > --- > > Key: DAFFODIL-1868 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1868 > Project: Daffodil > Issue Type: Bug > Components: TDML Runner >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > defaultConfig can be specified on the test suite. > This appears to only be checked (to see if there is a corresponding config) > if the test case has no config specified. It should be checked no matter what. > Second, defaultConfig doesn't accept name of an embedded config. > Lastly, it seems buggy. Not sure defaultConfig even worked. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DAFFODIL-1843) Unparser bitOrder change and OVC (outputValueCalc) interaction
[
https://issues.apache.org/jira/browse/DAFFODIL-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358869#comment-16358869
]
Dave Thompson commented on DAFFODIL-1843:
-
Correction: Latest updates were add8c5a6f82f25073475a4d6c7ae94fb5ca6e01a.
> Unparser bitOrder change and OVC (outputValueCalc) interaction
> --
>
> Key: DAFFODIL-1843
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1843
> Project: Daffodil
> Issue Type: Bug
> Components: Back End, Unparsing
>Affects Versions: 2.0.0
>Reporter: Michael Beckerle
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.1.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> See test test_bitOrderOVC1.
> This issue was discovered in the Link16 DFDL schema, and reproduced here in
> an isolated context.
> This test shows that a spurious Runtime SDE is detected due to a
> bitOrderChange. This has to do with the unparser having to back-up to fill in
> a suspension for an outputValueCalc element, and losing the bitOrder that
> should be maintained/captured for that usage.
> At least that's my theory.
> This test should work, but gets a Runtime SDE:
> {code}edu.illinois.ncsa.daffodil.tdml.TDMLException: Runtime Schema
> Definition Error: Data output stream DOS(id=5, Active, Direct Absolute from 0
> to 27 (length 27)) with bitOrder 'leastSignificantBitFirst' not on a byte
> boundary, cannot be populated from DOS(id=6, Finished, Buffered Absolute from
> 27 to 40 (length 13), data=1F00 no following)
> with bitOrder 'mostSignificantBitFirst'.
> Schema context: NNN Location line 64 column 18 in
> file:/tmp/s1_9132734461290850031.dfdl.xsd
> Data location was preceding byte 3
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1843) Unparser bitOrder change and OVC (outputValueCalc) interaction
[
https://issues.apache.org/jira/browse/DAFFODIL-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Thompson closed DAFFODIL-1843.
---
Pulled latest updates from incubator-daffodil
a1d881b13d26e47040c0302b2874cc78f0de27bf (v2.1.0-rc1).
Verified that scala-new no longer exists and the affected tests are now in
scala.
Verified that the affected tests still execute successfully.
> Unparser bitOrder change and OVC (outputValueCalc) interaction
> --
>
> Key: DAFFODIL-1843
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1843
> Project: Daffodil
> Issue Type: Bug
> Components: Back End, Unparsing
>Affects Versions: 2.0.0
>Reporter: Michael Beckerle
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.1.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> See test test_bitOrderOVC1.
> This issue was discovered in the Link16 DFDL schema, and reproduced here in
> an isolated context.
> This test shows that a spurious Runtime SDE is detected due to a
> bitOrderChange. This has to do with the unparser having to back-up to fill in
> a suspension for an outputValueCalc element, and losing the bitOrder that
> should be maintained/captured for that usage.
> At least that's my theory.
> This test should work, but gets a Runtime SDE:
> {code}edu.illinois.ncsa.daffodil.tdml.TDMLException: Runtime Schema
> Definition Error: Data output stream DOS(id=5, Active, Direct Absolute from 0
> to 27 (length 27)) with bitOrder 'leastSignificantBitFirst' not on a byte
> boundary, cannot be populated from DOS(id=6, Finished, Buffered Absolute from
> 27 to 40 (length 13), data=1F00 no following)
> with bitOrder 'mostSignificantBitFirst'.
> Schema context: NNN Location line 64 column 18 in
> file:/tmp/s1_9132734461290850031.dfdl.xsd
> Data location was preceding byte 3
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1868) TDML Runner defaultConfig cannot be embedded. Not checked properly.
[ https://issues.apache.org/jira/browse/DAFFODIL-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1868. --- Pulled latest updates from incubator-daffodil a1d881b13d26e47040c0302b2874cc78f0de27bf (v2.1.0-rc1). Verified that scala-new no longer exists and the affected tests are now in scala. Verified that the affected tests still execute successfully. > TDML Runner defaultConfig cannot be embedded. Not checked properly. > --- > > Key: DAFFODIL-1868 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1868 > Project: Daffodil > Issue Type: Bug > Components: TDML Runner >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > defaultConfig can be specified on the test suite. > This appears to only be checked (to see if there is a corresponding config) > if the test case has no config specified. It should be checked no matter what. > Second, defaultConfig doesn't accept name of an embedded config. > Lastly, it seems buggy. Not sure defaultConfig even worked. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DAFFODIL-1897) Some Schemas fail to compile with latest updates (v2.1.0-rc1)
Dave Thompson created DAFFODIL-1897: --- Summary: Some Schemas fail to compile with latest updates (v2.1.0-rc1) Key: DAFFODIL-1897 URL: https://issues.apache.org/jira/browse/DAFFODIL-1897 Project: Daffodil Issue Type: Bug Components: DFDL Schemas Affects Versions: 2.1.0 Reporter: Dave Thompson Attempted to execute the nightly test suite on incubator-daffodil commit add8c5a6f82f25073475a4d6c7ae94fb5ca6e01a. Four dfdl schemas failed to compile for the following formats: army-drrs, ato-rep, csvMixedNarrow and pcap. The command and error are provided below. Attempting to save parser for army-drrs. Test name is: army-drrs_10t_all_1000 CMD: /home/dfdl/incubator-daffodil/daffodil-cli/target/universal/stage/bin/daffodil -v save-parser -s /home/dfdl/ngf-dfdl/daffodil-perf/src/test/resources/edu/illinois/ncsa/daffodil/fouo_disa/army_drrs_lh/army_drrs_lh.dfdl.xsd > /home/dfdl/dfdl-testharness/saved_parsers/0.0.0/army-drrs.parser.bin process return code: 1 Failed to Save Parser - army-drrs Error: [error] Schema Definition Error: No schema document at location file:/home/dfdl/ngf-dfdl/daffodil-perf/src/test/resources/org/apache/daffodil/fouo_disa/army_drrs_lh/army_drrs_lh.dfdl.xsd. Schema context: Import Location in file:unknown [warning] Schema Definition Warning: schemaLocation property uses deprecated edu/illinois/ncsa/daffodil path instead of org/apache/daffodil. Converting to new path. Schema context: Import Location in file:unknown [info] Time (compiling): 219ms -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1872) Eclipse classpaths for daffodil-io module need fixing
[ https://issues.apache.org/jira/browse/DAFFODIL-1872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1872. --- Pulled latest updates from incubator-daffodil repository which included the associated commit, 44372a15de37c3c454e49eaa85f6ebff7bd0f04b. Verified by review that the code will only search daffodil-io/src/main. > Eclipse classpaths for daffodil-io module need fixing > - > > Key: DAFFODIL-1872 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1872 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Priority: Minor > Fix For: 2.1.0 > > > daffodil-io needs to include src/main/scala, but exclude src/test/scala not > just from the eclipse source directories but from the entire eclipse project. > Otherwise searching for things found in daffodil-io-unittest will always hit > twice. Once in daffodil-io, and again in daffodil-io-unittest. This can be > quite confusing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1866) Eclipse classpaths need fixing
[ https://issues.apache.org/jira/browse/DAFFODIL-1866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1866. --- Pulled latest updates from incubator-daffodil repository which included the specified commit, 4404af30e724c19be1eb1bf01bec8fd5a04600ea. Verified by review that the duplicate entries are no longer included in the .classpath files. > Eclipse classpaths need fixing > -- > > Key: DAFFODIL-1866 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1866 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Michael Beckerle >Priority: Major > Fix For: 2.1.0 > > Time Spent: 20m > Remaining Estimate: 0h > > UpdateEclipseClasspaths is putting duplicate entries into the generated > .classpath files that result in name conflicts. > Not sure why, but sbt compile updateClassifiers results in duplicate jars in > the lib_managed bundles and jars sub-dirs. > On my system, there is also a spurious duplication of the scala-xml library > (version 1.0.4 AND version 1.0.6 are found under > bundles/org.scala-lang.modules). It's not clear why this is happening. > Tool needs to be fixed to avoid the duplicates, and the .classpath files of > all the eclipse projects fixed to not have the duplicate entries. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1892) iCalendar performance to run tdml test is very slow
[ https://issues.apache.org/jira/browse/DAFFODIL-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1892. --- Closing in lieu of DAFFODIL-1893. > iCalendar performance to run tdml test is very slow > --- > > Key: DAFFODIL-1892 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1892 > Project: Daffodil > Issue Type: Bug > Components: DFDL Schemas >Affects Versions: 2.1.0 >Reporter: Michael Beckerle >Priority: Major > Fix For: 2.1.0 > > > The iCalendar schema (on DI2E.net) has been enhanced to have a couple unit > tests that use TDML runner. > These tests are tiny, yet on current 2.1.0 snapshot they take minutes to run. > Something is wrong. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (DAFFODIL-1878) Parse and unparse files/sec values vary significantly using compiled and saved parsers
[ https://issues.apache.org/jira/browse/DAFFODIL-1878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355603#comment-16355603 ] Dave Thompson commented on DAFFODIL-1878: - Additional information: Noticed that there were significant variances in the jpeg_10t_1m_1000 test between nightly runs, even when there was no change to daffodil when using saved parser. Looked at the raw csv data file and found that the average files/sec varied greatly between test iterations (e.g. 30 files/sec to 145 files/sec). Executed the test (ten iterations) numerous times varying the number files processed. Found that as the files processed was lowered the files/sec performance lowered as did the difference between the max and min files/sec lowered but remained significant. Executed with 5 threads and got similar results, lower files/sec and lower but still excessive difference. The difference when executing the test using the runtime compiled parser/schema was usually between 2 – 5 files/sec. > Parse and unparse files/sec values vary significantly using compiled and > saved parsers > -- > > Key: DAFFODIL-1878 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1878 > Project: Daffodil > Issue Type: Bug > Components: General >Affects Versions: 2.1.0 >Reporter: Dave Thompson >Assignee: Steve Lawrence >Priority: Major > Fix For: 2.2.0 > > Attachments: Performance report for 01-19-2018.docx > > > After updating the nightly scripts to use the pre-compiled/saved parsers the > result show that for some tests there are significant performance value > differences between using runtime compiled parsers and the saved parsers. > Attached is the report email. The Previous Val column is from non-saved > parser run. Curr Val column is from previously compiled/saved parsers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1732) ByteOrderChangeParser not inserted correctly for bitmap schema
[ https://issues.apache.org/jira/browse/DAFFODIL-1732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1732. --- Verified with latest daffodil and bitmap schema bmp files parse and unparse correctly. Resulting unparsed bmp file matches source file. > ByteOrderChangeParser not inserted correctly for bitmap schema > -- > > Key: DAFFODIL-1732 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1732 > Project: Daffodil > Issue Type: Bug > Components: Front End, General >Reporter: Steve Lawrence >Assignee: Michael Beckerle >Priority: Major > Fix For: 2.1.0 > > > The bitmap schema has a mixture of big and little endian byte orders. The > "Blob" elements are xs:hexBinary types with byteOrder="bigEndian". However, > they are being parsed as if they are littleEndian. Looking at the toBriefXML > parser, it looks like there are no ByteOrderChange parsers being inserted > before the hexBinaryBlob is parsed, and the most previous > ByteOrderChangeParser changed the byteOrder to littleEndian. So there appears > to be a bug where we aren't detecting all the places a ByteOrderChangeParser > is needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1841) daffodil root jar is published but shouldn't be, and its contents are wrong
[ https://issues.apache.org/jira/browse/DAFFODIL-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1841. --- Verified that the root jar and parent directory is no longer created when executing the sbt publishLocal command. > daffodil root jar is published but shouldn't be, and its contents are wrong > --- > > Key: DAFFODIL-1841 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1841 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Priority: Major > Fix For: 2.1.0 > > > Two issues here: > # running 'sbt publish' or 'sbt publishLocal' will publish a root jar, even > though we gave it the 'nopub' setting. > # the contents of the root jar are exactly the same as the daffodil-cli jar > (which is missing the 'nopub' setting). > The root jar shouldn't be published, and even if it were published, it should > not contain CLI classes. > We might want to look into upgrading to sbt 1.0 to resolve this problem and > switch to the newer sbt syntax. The current stuff is being deprecated if not > already removed in 1.0. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1876) Improve performance regression due to FormatInfo changes
[ https://issues.apache.org/jira/browse/DAFFODIL-1876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1876. --- Pulled latest updates from incubator-daffodil repository and verified the specified commit, ee4177b4c9bf84af2af19d8c9f67f283e233105e, was included. Reviewed updated files to determine the effected tests. Executed the entire SBT test suite. Verified that the effected tests executed successfully. Nightly test run executed successfully. Comparison with the FormatInfo commit (a1d881), showed performance gains across all data format tests with some exceptions. Some of these could be due to normal run-to-run deviation. There were no nato-stanag tests listed in this comparison due to these test being broken at the time. The comparison to the pre- FormatInfo commit (401761) showed that performance has not been regained to pre-FormatInfo commit values across all format tests. Most significant are the army-drss and nato-stanag tests which needs further investigation. Also note that for some tests there is a significant performance increase from the 2.0.0 baseline and in some there is still a significant performance decrease. Closing this ticket and created JIRA ticket DAFFODIL-1883 to identify the specific data formats that need further performance regression investigation. > Improve performance regression due to FormatInfo changes > > > Key: DAFFODIL-1876 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1876 > Project: Daffodil > Issue Type: Improvement > Components: Back End, Middle End >Affects Versions: 2.1.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > FormatInfo changes introduced significant performance degradation. > Changes are needed to win back this performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (DAFFODIL-1883) Investigate remaining performance regression issues
Dave Thompson created DAFFODIL-1883: --- Summary: Investigate remaining performance regression issues Key: DAFFODIL-1883 URL: https://issues.apache.org/jira/browse/DAFFODIL-1883 Project: Daffodil Issue Type: Bug Components: Performance Affects Versions: 2.1.0 Reporter: Dave Thompson Fix For: 2.2.0 While verifying JIRA ticket DAFFODIL-1876, found that although the majority of the performance degradation from the FormatInfo changes (commit a1d881) has been recovered, there is still performance regression on several data format tests. The most significant performance regression occurs with formats army-drss, asterix and nato-stanag tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1739) Implement 'packed' and 'bcd'
[ https://issues.apache.org/jira/browse/DAFFODIL-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1739. --- Pulled latest updates from incubator-daffodil repository and verified the specified commit, 8d9aadab07a2f422dfae527b3a4d1cb4047a6efa was included. Reviewed updated files to determine the effected tests. Verified that the effected tests executed successfully with the exception of referenced test "length_delimited_12_05". This test was still in scala-debug folder and when executed it failed. Talked with Josh A. and he said that it failed due needing support for delimited parsing of non ISO-8859-1 encodings which is not currently implemented. The issue is addressed by DAFFODIL-1541. As we were investigating I asked about another test, simple_type_properties_binary_number_13_01, that was in both the scala and scala-debug test files. Josh removed the entry from the scala-debug test file. Verified that the test was removed for scala-debug in commit 11d36b21e1abb7fc6fd26f7aa10cab3827f53546. > Implement 'packed' and 'bcd' > > > Key: DAFFODIL-1739 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1739 > Project: Daffodil > Issue Type: New Feature > Components: DFDL Language >Reporter: Elizabeth Fahl >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > Implement 'packed' and 'bcd' and for binaryNumberRep and binaryCalendarRep. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1597) Too many ways that encoding, byteOrder, etc. are being setup
[ https://issues.apache.org/jira/browse/DAFFODIL-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1597. --- Pulled latest updates from incubator-daffodil repository and verified the specified commit, ee4177b4c9bf84af2af19d8c9f67f283e233105e, was included. Reviewed updated files to determine the effected tests. Executed the entire SBT test suite. Verified that the effected tests executed successfully. > Too many ways that encoding, byteOrder, etc. are being setup > > > Key: DAFFODIL-1597 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1597 > Project: Daffodil > Issue Type: Improvement > Components: Back End, Clean Ups >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > This issue affects both parser and unparser. > It is a code-cleanup/maintainability, and perhaps a bit of a performance > issue. > I'll use encoding as the property for discussion here, but the point applies > to byteOrder and possibly bitOrder and maybe a few other things like fillByte > also. > We have this idea that since encoding often doesn't change at all in a data > format, that we insert an encoding-change processor (parser or unparser) > which just sets the new setting, which is then stored in the > DataInputStream/DataOutputStream object, and no per-element overhead is > encountered for setting up this property. > For some formats, encoding changes. This becomes quite tricky if the change > occurs inside of a repeating element, as in that case, each repeat might need > to begin by setting the encoding to the right one for the start of the > repeating element, but then inside that element another set might change it, > and that might last until the end of the repeating element, hence, at the > start of the next repeat, we must set it back to the proper encoding for the > start of the element. > So, unless the DFDL compiler can optimize it out, any repeating element > containing any text must begin with an encodingChange processor. Right now > the optimization looks for whether the entire schema has uniform encoding > (which is common), but any format that has a runtime-valued expression for > encoding will be deemed "unknowable", and the encoding will be assumed to be > variable. This is very pessimistic, but a better analysis depends on > determining that a runtime-valued expression for encoding is still defined at > a high-enough scope (such as top-level) that encoding is not > subject-to-change during the element in question. The analysis being done is > not that sophisticated currently. > The above might not matter much for encoding, but for byteOrder, where we > know there are formats that begin with a byte-order indication (such as > PCAP), this matters. > Now, there are also some parsers (and unparser) primitive processors that > expliclity set encoding, or byteOrder. before they carry out their > processing. > It is possible that encoding change processors are not being inserted > everywhere they are needed; hence, if the various setEncoding calls are > removed, it may break things. > Now, setting the encoding can check if the encoding is the same one it can do > little work other than an equality check. So rather than having lots of > encoding-change processors that are being inserted due to unsophisticated > compile-time optimization, it may actually be better performance to either > simply call setEncoding before every textual primitive operation, or it may > be better to use Evaluatables, and have the data stream layer access encoding > information via the Evaluatables mechanism. Then at least we would have only > one code path to focus on for performance improvement. > For unparsing: > slightly more complicated - any suspended operation needs to "freeze" the > value of encoding that is used to unparse, so that subsequent non-suspended > unparser operations that change the encoding will not result in the suspended > operation using the wrong one. > However, since the encoding change unparser will have been run before the > suspension is created, and since the data output stream of a suspension is > cloned for the suspension, the encoding should be correctly set for unparsers > that are suspended. *Should* being the key operative word here. This needs to > be verified. (Also, the cloning of the data output stream state is itself a > big performance worry, so the work done there needs to be minimized.) > Whatever mechanism is chosen to resolve the parser issue, the unparser should > work the same way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1001) Invalid bitOrder and byteOrder combination should produce SDE
[ https://issues.apache.org/jira/browse/DAFFODIL-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1001. --- Verified the latest updates from the incubator-daffodil repository included the specified commit, ee4177b4c9bf84af2af19d8c9f67f283e233105e. Verified that the specified test executes successfully. > Invalid bitOrder and byteOrder combination should produce SDE > - > > Key: DAFFODIL-1001 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1001 > Project: Daffodil > Issue Type: Bug > Components: Diagnostics, Front End >Affects Versions: s14 >Reporter: Elizabeth Fahl >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > In the document for bitOrder at [http://redmine.ogf.org/dmsf_files/13309], > section 3.2 states that there are only 3 valid combinations of bitOrder and > byteOrder and other combinations should produce a schema definition error. I > created a test with byteOrder of 'bigEndian' and bitOrder of > 'leastSignificantBitFirst' but the test parses without errors. > See test 'test_bigEndianLeastFirst' in > daffodil-test/src/test/scala-debug/edu/illinois/ncsa/daffodil/section05/simple_types/TestSimpleTypes2.scala > and daffodil-cd > test/src/test/resources/edu/illinois/ncsa/daffodil/section05/simple_types/BitOrder.tdml -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (DAFFODIL-1001) Invalid bitOrder and byteOrder combination should produce SDE
[ https://issues.apache.org/jira/browse/DAFFODIL-1001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson updated DAFFODIL-1001: Description: In the document for bitOrder at [http://redmine.ogf.org/dmsf_files/13309], section 3.2 states that there are only 3 valid combinations of bitOrder and byteOrder and other combinations should produce a schema definition error. I created a test with byteOrder of 'bigEndian' and bitOrder of 'leastSignificantBitFirst' but the test parses without errors. See test 'test_bigEndianLeastFirst' in daffodil-test/src/test/scala-debug/edu/illinois/ncsa/daffodil/section05/simple_types/TestSimpleTypes2.scala and daffodil-cd test/src/test/resources/edu/illinois/ncsa/daffodil/section05/simple_types/BitOrder.tdml was: In the document for bitOrder at [http://redmine.ogf.org/dmsf_files/13309], section 3.2 states that there are only 3 valid combinations of bitOrder and byteOrder and other combinations should produce a schema definition error. I created a test with byteOrder of 'bigEndian' and bitOrder of 'leastSignificantBitFirst' but the test parses without errors. See test 'test_bigEndianLeastFirst' in daffodil-test/src/test/scala-debug/edu/illinois/ncsa/daffodil/section05/simple_types/TestSimpleTypes2.scala and daffodil-test/src/test/resources/edu/illinois/ncsa/daffodil/section05/simple_types/BitOrder.tdml > Invalid bitOrder and byteOrder combination should produce SDE > - > > Key: DAFFODIL-1001 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1001 > Project: Daffodil > Issue Type: Bug > Components: Diagnostics, Front End >Affects Versions: s14 >Reporter: Elizabeth Fahl >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > In the document for bitOrder at [http://redmine.ogf.org/dmsf_files/13309], > section 3.2 states that there are only 3 valid combinations of bitOrder and > byteOrder and other combinations should produce a schema definition error. I > created a test with byteOrder of 'bigEndian' and bitOrder of > 'leastSignificantBitFirst' but the test parses without errors. > See test 'test_bigEndianLeastFirst' in > daffodil-test/src/test/scala-debug/edu/illinois/ncsa/daffodil/section05/simple_types/TestSimpleTypes2.scala > and daffodil-cd > test/src/test/resources/edu/illinois/ncsa/daffodil/section05/simple_types/BitOrder.tdml -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1846) TDML Runner - Infoset.contents is stripping off all attributes
[ https://issues.apache.org/jira/browse/DAFFODIL-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1846. --- Verified the latest updates from the incubator-daffodil repository included the specified commit, 23fbbfa1a10e707411d0d12bc01d8532fda71b33. Verified that the specified tests execute successfully. > TDML Runner - Infoset.contents is stripping off all attributes > -- > > Key: DAFFODIL-1846 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1846 > Project: Daffodil > Issue Type: Bug > Components: TDML Runner >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > The TDML runner removes all attributes from the Infoset XML that it pulls in > from the TDML file. This can result in unparser errors due to loss of > namespace bindings. > The Infoset.contents call redundantly calls the utility to remove attributes > more than once. > This needs to be cleaned up and is related to DFDL-1384. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1864) spurious SDE about all-text schema being non-scannable
[ https://issues.apache.org/jira/browse/DAFFODIL-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1864. --- Will investigate if additional tests are needed as part of an over all review of test coverage, when time permits. > spurious SDE about all-text schema being non-scannable > -- > > Key: DAFFODIL-1864 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1864 > Project: Daffodil > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Blocker > Fix For: 2.0.1 > > Time Spent: 1h > Remaining Estimate: 0h > > The isScannable computation is incorrect. For entirely textual formats it > still comes up with isScannable false. > This causes SDE if you are using lengthKind pattern. The SDE says the data > must be scannable in order to use lengthKind pattern. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1860) Minimize the number of HashMap allocations/insertions in the Infoset
[ https://issues.apache.org/jira/browse/DAFFODIL-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1860. --- Verified the latest updates from the incubator-daffodil repository included the specified commit, eec6c76a36edeecebf87ab5a3d80f9040442f120. Verified all SBT tests for effected subprojects execute successfully. Verified a performance increase across most data types of from a slight increase up to 46%. Note that a few tests across data formats show a slight decrease, within normal nightly deviation. > Minimize the number of HashMap allocations/insertions in the Infoset > > > Key: DAFFODIL-1860 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1860 > Project: Daffodil > Issue Type: Bug > Components: Back End >Reporter: Steve Lawrence >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Performance testing found that the slot removal made in DAFFODIL-1854 caused > performance degradation, sometimes up to 20% decreases. Profiling found that > some of this is due to the allocations of the HashMap used for quickly > looking up elements in the Infoset by QName. We should only allocate hashmaps > when it's possible an element could be be used in an expression, which should > improve performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1773) Choice ambiguous element name results in failed expression
[
https://issues.apache.org/jira/browse/DAFFODIL-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Thompson closed DAFFODIL-1773.
---
Verified the latest updates from the incubator-daffodil repository included the
specified commit, 02c8324294ed4395c1231c6ea3ec6ae77139ba46.
Verified the specified tests execute successfully.
> Choice ambiguous element name results in failed expression
> --
>
> Key: DAFFODIL-1773
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1773
> Project: Daffodil
> Issue Type: Bug
> Components: Middle End
>Reporter: Michael Beckerle
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.1.0
>
>
> (Bug found in Link16 work. Reproduced in tests test_choiceSlotAmbiguous1 and
> test_choiceSlotAmbiguous2)
> Consider this choice:
> {code:java}
>
> ��
> �� dfdl:length="1" />
> �� dfdl:length="1">
> ��
> �� source="http://www.ogf.org/dfdl;>
> �� { ../A eq
> "A" }
> ��
> ��
> ��
> ��
> ��
> �� dfdl:length="1" />
> �� dfdl:length="1">
> ��
> �� source="http://www.ogf.org/dfdl;>
> �� { ../B eq
> "B" }
> ��
> ��
> ��
> ��
> �� {code}
> Now imagine a subsequent expression containing ../C.
> Which C is that? The first or second. Answer is it depends on which
> discriminator was chosen.
> If the first discriminator is true, then the path ../C succeeds. If the
> second discriminator was true the path ../C fails with no such element C.
> This is probably due to Daffodil's schema compiler assigning two different
> slot numbers to these two C elements, rather than recognizing they have the
> same name+namespace and so using a single slot for them.
> ��
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DAFFODIL-1843) Unparser bitOrder change and OVC (outputValueCalc) interaction
[
https://issues.apache.org/jira/browse/DAFFODIL-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337597#comment-16337597
]
Dave Thompson commented on DAFFODIL-1843:
-
Verified the latest updates from the incubator-daffodil repository included the
specified commit, a1d881b13d26e47040c0302b2874cc78f0de27bf. Currently the
specified test is in scala-new which is no longer get checked during test. This
causes no test to run messages. Temporarily copied the scala test file to the
associated scala folder and ran the test. The test executed successfully. Steve
said a future update will correct the scala-new issue. Will rerun the tests
when that update is committed.
> Unparser bitOrder change and OVC (outputValueCalc) interaction
> --
>
> Key: DAFFODIL-1843
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1843
> Project: Daffodil
> Issue Type: Bug
> Components: Back End, Unparsing
>Affects Versions: 2.0.0
>Reporter: Michael Beckerle
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.1.0
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> See test test_bitOrderOVC1.
> This issue was discovered in the Link16 DFDL schema, and reproduced here in
> an isolated context.
> This test shows that a spurious Runtime SDE is detected due to a
> bitOrderChange. This has to do with the unparser having to back-up to fill in
> a suspension for an outputValueCalc element, and losing the bitOrder that
> should be maintained/captured for that usage.
> At least that's my theory.
> This test should work, but gets a Runtime SDE:
> {code}edu.illinois.ncsa.daffodil.tdml.TDMLException: Runtime Schema
> Definition Error: Data output stream DOS(id=5, Active, Direct Absolute from 0
> to 27 (length 27)) with bitOrder 'leastSignificantBitFirst' not on a byte
> boundary, cannot be populated from DOS(id=6, Finished, Buffered Absolute from
> 27 to 40 (length 13), data=1F00 no following)
> with bitOrder 'mostSignificantBitFirst'.
> Schema context: NNN Location line 64 column 18 in
> file:/tmp/s1_9132734461290850031.dfdl.xsd
> Data location was preceding byte 3
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DAFFODIL-1868) TDML Runner defaultConfig cannot be embedded. Not checked properly.
[ https://issues.apache.org/jira/browse/DAFFODIL-1868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337563#comment-16337563 ] Dave Thompson commented on DAFFODIL-1868: - Verified the latest updates from the incubator-daffodil repository included the specified commit, a1d881b13d26e47040c0302b2874cc78f0de27bf. Currently the specified tests are in scala-new which is no longer get checked during test. This causes no test to run messages. Temporarily copied the scala test file to the associated scala folder and ran the test suite. The tests executed successfully. Steve said a future update will correct the scala-new issue. Will rerun the tests when that update is committed. > TDML Runner defaultConfig cannot be embedded. Not checked properly. > --- > > Key: DAFFODIL-1868 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1868 > Project: Daffodil > Issue Type: Bug > Components: TDML Runner >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Major > Fix For: 2.1.0 > > > defaultConfig can be specified on the test suite. > This appears to only be checked (to see if there is a corresponding config) > if the test case has no config specified. It should be checked no matter what. > Second, defaultConfig doesn't accept name of an embedded config. > Lastly, it seems buggy. Not sure defaultConfig even worked. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1869) Nato link16 doesn't work on 2.0.0 Daffodil
[ https://issues.apache.org/jira/browse/DAFFODIL-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1869. --- Verified the latest updates from the incubator-daffodil repository included the specified commit, 02c8324294ed4395c1231c6ea3ec6ae77139ba46. Verified the automatic data pull script pulled the latest update to the nato-stanag repository which included the associated commit. Executed the nightly nato-stanag tests, which were failing due to previous daffodil updates. The parse test now pass as expected. The unparse tests failed due to changes in the nato-stanag schema which rendered the existing infoset test files incorrect. Re-parsed the original test data files recreating the infoset test files. Reran the nightly nato-stanag unparse test and they now pass as expected. > Nato link16 doesn't work on 2.0.0 Daffodil > -- > > Key: DAFFODIL-1869 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1869 > Project: Daffodil > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Blocker > Fix For: 2.1.0 > > > From NATO NCIA: > Are you still planning to release a patch? The last Daffodil revision that > was working is 3e754f259cbe9c36d96379921bc060acc4f8ab24 (23rd of May 2017). > In later revisions, the compilation of the Link 16 schema is going wrong at > J7.5I regarding the choice on "ActionIffSifManagement_1606-006 and the choice > on "ModeICodeApplicability_1610-001". > Reproducing this requires access to the Link16 schema, which requires access > to DI2E.net DFDL/Daffodil project. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (DAFFODIL-1612) sbt - branches without tags cause sbt compile to fail
[ https://issues.apache.org/jira/browse/DAFFODIL-1612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1612. --- As this ticket's issue has been addressed and verified in JIRA ticket DAFFODIL-1856, closing this ticket. > sbt - branches without tags cause sbt compile to fail > - > > Key: DAFFODIL-1612 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1612 > Project: Daffodil > Issue Type: Improvement > Components: Infrastructure >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Priority: Minor > Fix For: 2.1.0 > > > So if you happen to have a git repository with just say, the 2.0.0 branch on > it, which has no tags as of yet, then if you try to do sbt compile it fails > to resolve dependencies because the git logic which tries to determine > whether you are a snapshot or a tag fails. > This is easy to work around, add any tag, or clone the whole repository, not > just one branch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (DAFFODIL-1873) sbt warnings/errors
[
https://issues.apache.org/jira/browse/DAFFODIL-1873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dave Thompson closed DAFFODIL-1873.
---
Verified the specified warning and error messages occur prior to specified the
specified commit.
Pulled the latest updates from incubator-daffodil repository which included the
specified commit 40176161e67484f3bebd73ffd2639f1645feffd3.
Executed the specified commands and verified that the warning and error
messages no longer occur.
> sbt warnings/errors
>
>
> Key: DAFFODIL-1873
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1873
> Project: Daffodil
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Michael Beckerle
>Assignee: Steve Lawrence
> Fix For: 2.1.0
>
>
> I did these commands to do a completely clean build:
> {code}
> git checkout master
> git clean -xdf
> sbt compile
> sbt updateClassifiers
> {code}
> It produces quite a few error/warning messages that are troubling. If these
> can be eliminated or even reduced it will be much less confusing for new
> developers.
> {code}
> [info] Loading settings from plugins.sbt ...
> [info] Loading project definition from
> /home/mbeckerle-unencrypted/daffodil/project
> [info] Updating
> {file:/home/mbeckerle-unencrypted/daffodil/project/}daffodil-build...
> [info] Done updating.
> [warn] Found version conflict(s) in library dependencies; some are suspected
> to be binary incompatible:
> [warn]* org.codehaus.plexus:plexus-utils:3.0.17 is selected over
> {2.1, 1.5.5}
> [warn]+- org.apache.maven:maven-settings:3.2.2
> (depends on 3.0.17)
> [warn]+- org.apache.maven:maven-repository-metadata:3.2.2
> (depends on 3.0.17)
> [warn]+- org.apache.maven:maven-aether-provider:3.2.2
> (depends on 3.0.17)
> [warn]+- org.apache.maven:maven-model:3.2.2
> (depends on 3.0.17)
> [warn]+- org.apache.maven:maven-core:3.2.2
> (depends on 3.0.17)
> [warn]+- org.apache.maven:maven-artifact:3.2.2
> (depends on 3.0.17)
> [warn]+- org.apache.maven:maven-settings-builder:3.2.2
> (depends on 3.0.17)
> [warn]+- org.apache.maven:maven-model-builder:3.2.2
> (depends on 3.0.17)
> [warn]+- org.sonatype.plexus:plexus-sec-dispatcher:1.3
> (depends on 1.5.5)
> [warn]+- org.eclipse.sisu:org.eclipse.sisu.plexus:0.0.0.M5
> (depends on 2.1)
> [warn]* com.google.guava:guava:18.0 is selected over {10.0.1, 15.0}
> [warn]+- com.spotify:docker-client:3.5.13
> (depends on 18.0)
> [warn]+-
> com.fasterxml.jackson.datatype:jackson-datatype-guava:2.6.0 (depends on 15.0)
> [warn]+- org.eclipse.sisu:org.eclipse.sisu.plexus:0.0.0.M5
> (depends on 10.0.1)
> [warn] Run 'evicted' to see detailed eviction warnings
> [info] Compiling 1 Scala source to
> /home/mbeckerle-unencrypted/daffodil/project/target/scala-2.12/sbt-1.0/classes
> ...
> [info] Done compiling.
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Loading settings from build.sbt ...
> [info] Resolving key references (10042 settings) ...
> [info] Set current project to daffodil (in build
> file:/home/mbeckerle-unencrypted/daffodil/)
> [info] Updating {file:/home/mbeckerle-unencrypted/daffodil/}daffodil...
> [info] Updating
> {file:/home/mbeckerle-unencrypted/daffodil/}daffodil-propgen...
> [info] Updating
> {file:/home/mbeckerle-unencrypted/daffodil/}daffodil-macro-lib...
> [info] Done updating.
> [info] Done updating.
> [info] Done updating.
> [info] Updating {file:/home/mbeckerle-unencrypted/daffodil/}daffodil-lib...
> [info] Done updating.
> [info] Updating {file:/home/mbeckerle-unencrypted/daffodil/}daffodil-io...
> [info] Done updating.
> [info] Updating
> {file:/home/mbeckerle-unencrypted/daffodil/}daffodil-runtime1...
> [error] a required artifact is not listed by module descriptor: *#*!*.*
> [error] a required artifact is not listed by module descriptor: *#*!*.*
> [error] a required artifact is not listed by module descriptor: *#*!*.*
> [error] a required artifact is not listed by module descriptor: *#*!*.*
> [error] a required artifact is not listed by module descriptor: *#*!*.*
> [error] a required artifact is not listed by module descriptor: *#*!*.*
> [info] Done updating.
> [info] Updating
> {file:/home/mbeckerle-unencrypted/daffodil/}daffodil-runtime1-unparser...
> [error] a required artifact is not listed by module descriptor: *#*!*.*
> [error] a required artifact is not listed by module descriptor: *#*!*.*
> [error] a required artifact is not listed by module descriptor: *#*!*.*
> [error] a required artifact is
[jira] [Closed] (DAFFODIL-1862) Better logging/reset mechanism for MarkPool
[ https://issues.apache.org/jira/browse/DAFFODIL-1862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1862. --- pulled latest updates for incubator-daffodil which included the specified commit, 541bb07801d44e976d2feb256cc04735268445b0. Reviewed the code and talked with dev and determined that it would be very difficult to simulate a condition that would trigger a markpool leak. Determined that the code reviews would be sufficient in this case. Verified that the system successfully built and executed the sbt tests. Nightly run also executed successfully. > Better logging/reset mechanism for MarkPool > --- > > Key: DAFFODIL-1862 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1862 > Project: Daffodil > Issue Type: Bug > Components: Back End >Reporter: Steve Lawrence >Assignee: Steve Lawrence > Fix For: 2.1.0 > > Time Spent: 20m > Remaining Estimate: 0h > > It is possible for MarkPool instances to leak, which often be difficult to > track down and can lead to denial of service with API users. We need a > mechanism to ensure MarkPools instances are never leaked, and to also > discover bugs that would cause them to leak. > See the folloiwng thread for a more detailed discussion of the problems: > https://lists.apache.org/thread.html/914d5e7e0e4bf9001cb048034608b1b6bcffa04de9a10158302e@%3Cdev.daffodil.apache.org%3E -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (DAFFODIL-1864) spurious SDE about all-text schema being non-scannable
[ https://issues.apache.org/jira/browse/DAFFODIL-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson reassigned DAFFODIL-1864: --- Assignee: Dave Thompson (was: Michael Beckerle) > spurious SDE about all-text schema being non-scannable > -- > > Key: DAFFODIL-1864 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1864 > Project: Daffodil > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Michael Beckerle >Assignee: Dave Thompson >Priority: Blocker > Fix For: 2.0.1 > > Time Spent: 1h > Remaining Estimate: 0h > > The isScannable computation is incorrect. For entirely textual formats it > still comes up with isScannable false. > This causes SDE if you are using lengthKind pattern. The SDE says the data > must be scannable in order to use lengthKind pattern. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (DAFFODIL-1851) Unparsing iCalendar files appears to drop data that is included in the infoset file
[ https://issues.apache.org/jira/browse/DAFFODIL-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1851. --- Pulled lasts updates from incubator-daffodil repository which included the specified commit. Verified that unparsing an iCalendar file no longer truncates/drops information. Note: there is an wrapping issue in the unparsed file which is addressed in JIRA ticket DAFFODIL-1734. > Unparsing iCalendar files appears to drop data that is included in the > infoset file > --- > > Key: DAFFODIL-1851 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1851 > Project: Daffodil > Issue Type: Bug > Components: Unparsing >Affects Versions: 2.0.0 >Reporter: Dave Thompson >Assignee: Dave Thompson > Fix For: 2.1.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Although the smaller iCalendar files appear to parse and unparse > successfully, the resulting unparsed files are missing much of the calendar > data. Below is BeyondCompare Text comparison of the source iCal file and the > unparsed iCal file. Source is on the left. > !image-2017-10-04-09-48-16-226.png! > The parsed (infoset) files appear to include all of the data missing from the > unparsed file. > Attached the source, infoset and unparsed files for one iCalendar file. > �� -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (DAFFODIL-1854) Remove concept of slots from InfosetImpl.scala
[ https://issues.apache.org/jira/browse/DAFFODIL-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson closed DAFFODIL-1854. --- Pulled branch daffodil branch 2.1.0. Executed all tests under daffodil-test which included the specified tests added (4 test with one commented out) to verify. All of these daffodil-test tests including the 3 active tests passed. Talked with Steve and he felt that the commented test should currently pass as is. I uncommented the test, executed it and found that the test did not pass as expected. JIRA ticket DAFFODIL-1865 was generated to address the issue causing test test_array_IndexOutOfBounds_04 to fail. Closing this ticket as the identified issue being fixed. > Remove concept of slots from InfosetImpl.scala > -- > > Key: DAFFODIL-1854 > URL: https://issues.apache.org/jira/browse/DAFFODIL-1854 > Project: Daffodil > Issue Type: Improvement > Components: Back End, Clean Ups >Affects Versions: 2.1.0 >Reporter: Taylor Wise >Assignee: Dave Thompson > Fix For: 2.1.0 > > Time Spent: 2h 5m > > Remove the concept of 'slots' from InfosetImpl.scala. ��One of a series of > steps towards resolving space compilation issues and also re-enabling > unordered sequences. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
