[jira] [Resolved] (DAFFODIL-2895) Runtime SDE in ElementCombinator parser with validation causes "Expression Evaluation Error: Element does not have a value"
[ https://issues.apache.org/jira/browse/DAFFODIL-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2895. -- Fix Version/s: 3.8.0 Resolution: Fixed Fixed in commit b2aa98d9648c69c9a4e518b3826ef25f550ec786 > Runtime SDE in ElementCombinator parser with validation causes "Expression > Evaluation Error: Element does not have a value" > --- > > Key: DAFFODIL-2895 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2895 > Project: Daffodil > Issue Type: Bug > Components: Back End >Affects Versions: 3.7.0 >Reporter: Steve Lawrence >Priority: Major > Fix For: 3.8.0 > > > The finally block for the ElementCombinator parser calls parseEnd which does > restriction validation. But if an exception was thrown then we do not have a > value to validate, which creates an "Element does not have a value" exception > and hides the underlying exception. We should only perform validation if the > parser actually succeeds, so this logic needs to be moved out of the finally > block. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2896) validationMode=full enables Daffodils limited validation
Steve Lawrence created DAFFODIL-2896: Summary: validationMode=full enables Daffodils limited validation Key: DAFFODIL-2896 URL: https://issues.apache.org/jira/browse/DAFFODIL-2896 Project: Daffodil Issue Type: Bug Components: Back End Reporter: Steve Lawrence Fix For: 3.7.0 Daffodil has three validation modes: off, limited, and full. "Limited" enables Daffodils internal validation during parsing and "full" enables Xerces validation at the end of parsing. However, "full" also enables Daffodils limited validation, so we incur extra overhead with full validation. We should change full validation so it only does Xerces validation. Note that this change is not backwards compatible and could potentially break tests that use full validation but specifically look for Daffodil validation messages. Alternatively, we may want to consider adding a new validation mode that only enables xerces. Then "full" could keep its current behavior of validating with Daffodil and Xerces. This has the added benefit that we can get complete test coverage of both validation mechanisms without having to duplicate tests. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DAFFODIL-2895) Runtime SDE in ElementCombinator parser with validation causes "Expression Evaluation Error: Element does not have a value"
[ https://issues.apache.org/jira/browse/DAFFODIL-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence updated DAFFODIL-2895: - Component/s: Back End Affects Version/s: 3.7.0 Description: The finally block for the ElementCombinator parser calls parseEnd which does restriction validation. But if an exception was thrown then we do not have a value to validate, which creates an "Element does not have a value" exception and hides the underlying exception. We should only perform validation if the parser actually succeeds, so this logic needs to be moved out of the finally block. Issue Type: Bug (was: Improvement) Summary: Runtime SDE in ElementCombinator parser with validation causes "Expression Evaluation Error: Element does not have a value" (was: Runtime SDE in ElementCombinator parser with validation causes Expression Evaluation Error: Element does not have a value) > Runtime SDE in ElementCombinator parser with validation causes "Expression > Evaluation Error: Element does not have a value" > --- > > Key: DAFFODIL-2895 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2895 > Project: Daffodil > Issue Type: Bug > Components: Back End >Affects Versions: 3.7.0 >Reporter: Steve Lawrence >Priority: Major > > The finally block for the ElementCombinator parser calls parseEnd which does > restriction validation. But if an exception was thrown then we do not have a > value to validate, which creates an "Element does not have a value" exception > and hides the underlying exception. We should only perform validation if the > parser actually succeeds, so this logic needs to be moved out of the finally > block. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2895) Runtime SDE in ElementCombinator parser with validation causes Expression Evaluation Error: Element does not have a value
Steve Lawrence created DAFFODIL-2895: Summary: Runtime SDE in ElementCombinator parser with validation causes Expression Evaluation Error: Element does not have a value Key: DAFFODIL-2895 URL: https://issues.apache.org/jira/browse/DAFFODIL-2895 Project: Daffodil Issue Type: Improvement Reporter: Steve Lawrence -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2643) unit test needed that shows how to capture and examine log for proper logging
[
https://issues.apache.org/jira/browse/DAFFODIL-2643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840431#comment-17840431
]
Steve Lawrence commented on DAFFODIL-2643:
--
Once this is implemented and tests are able to capture logs, we should make it
so that if a test fails we output the captured log information in addition to
the normal diagnostics. Sometimes the test failure isn't clear and it's
possible that additional log information might help.
> unit test needed that shows how to capture and examine log for proper logging
> -
>
> Key: DAFFODIL-2643
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2643
> Project: Daffodil
> Issue Type: Bug
> Components: Infrastructure, QA
>Affects Versions: 3.2.1
>Reporter: Mike Beckerle
>Priority: Minor
>
> Proper logging should be testable.
> We need a unit test that sets up a logger and verifies that a particular
> logging message is present. The purpose of this test is (a) show logging is
> working at all (b) show how to do this so as to insure certain log messages
> are present.
> Ideally the TDML language should have a way of specifying expected log output:
> {code:java}
>
> string here that must be in log message case insensitive
> another such string
> {code}
> And consistent with DAFFODIL-864, which calls for TDML negative errors,
> warnings, validation errors, etc. there should be negative log checking as
> well.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2894) Stabalize and document pluggable charset API
[
https://issues.apache.org/jira/browse/DAFFODIL-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840424#comment-17840424
]
Steve Lawrence commented on DAFFODIL-2894:
--
Included in this should be improved error messages. For example, if you create
a custom charset that does not have the correct number of characters in the
decodeString, the base class throws an assertion exception, something like:
{code:java}
org.apache.daffodil.lib.exceptions.UsageException: Usage error:
BitsCharsetNonByteSize.this.decodeString.length().==(1.<<(BitsCharsetNonByteSize.this.bitWidthOfACodeUnit))
at org.apache.daffodil.lib.exceptions.Assert$.usageError(Assert.scala:139)
at
org.apache.daffodil.io.processors.charset.BitsCharsetNonByteSize.$init$(BitsCharsetNonByteSize.scala:67)
{code}
However, when this is thrown it is caught by the service loader, which hides
this exception and just creates a warning log that says
{code:java}
[warn] Named service BitsCharsetDefinition failed to load. Cause:
org.apache.daffodil.io.processors.charset.BitsCharsetDefinition: Provider
com.example.CustomCharset could not be instantiated{code}
This is not helpful at all. For example, this really needs to include the
exception that prevented the charset from being instantiated, and the assertion
probably needs to be more human readable.
> Stabalize and document pluggable charset API
>
>
> Key: DAFFODIL-2894
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2894
> Project: Daffodil
> Issue Type: Bug
> Components: API
>Reporter: Steve Lawrence
>Priority: Major
> Fix For: 4.0.0
>
>
> Daffodil supports the ability to have pluggable character sets. This is
> supported, but the API is not really documented and is not part of the
> "public" api. There aren't many known external charsets, but it's possible
> that might change. Like we did with Layers, we may want to take another look
> at our character set API and see if it is sufficient move it from an
> experimental API to a more public stable API.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2892) JAR file URI processing does not allow for JARs within JARs
[
https://issues.apache.org/jira/browse/DAFFODIL-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17840393#comment-17840393
]
Steve Lawrence commented on DAFFODIL-2892:
--
We don't generally do hotfixes except for security issues. But the development
cycle for 3.8.0 is expected to be relatively short, hopefully releasing
sometime around late May.
> JAR file URI processing does not allow for JARs within JARs
> ---
>
> Key: DAFFODIL-2892
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2892
> Project: Daffodil
> Issue Type: Bug
> Components: General
>Affects Versions: 3.6.0, 3.7.0
> Environment: OpenJDK Runtime Environment Temurin-11.0.22+7 (build
> 11.0.22+7)
>Reporter: Simon Godden
>Assignee: Steve Lawrence
>Priority: Critical
> Fix For: 3.8.0
>
>
> Code was introduced in version 3.6.0 which assumes that all JAR file URIs are
> composed of two parts - the path to the JAR file, and the path to the
> resource within the JAR file, separated by a an exclamation mark.
> This means that deployments using "uber" jars now fail if the schema is
> located inside a JAR inside BOOT-INF/lib within the top level JAR.
> A microservice deployment with a schema URI such as this fails:
> {{jar:file:/home/serviceuser/service.jar!/BOOT-INF/lib/some-other-jar.jar!/some-schema.dfdl.xsd}}
> The code at
> [https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala#L185
>
> |https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala#L185]then
> fails on the assertion that there are 2 parts after it splits by the
> separator.
> Versions up to 3.5.0 work fine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2892) JAR file URI processing does not allow for JARs within JARs
[
https://issues.apache.org/jira/browse/DAFFODIL-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2892.
--
Resolution: Fixed
Fixed in commit a853a2fc6d06ad393071b3df48bbfc8d9a01ebec
> JAR file URI processing does not allow for JARs within JARs
> ---
>
> Key: DAFFODIL-2892
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2892
> Project: Daffodil
> Issue Type: Bug
> Components: General
>Affects Versions: 3.6.0, 3.7.0
> Environment: OpenJDK Runtime Environment Temurin-11.0.22+7 (build
> 11.0.22+7)
>Reporter: Simon Godden
>Assignee: Steve Lawrence
>Priority: Critical
> Fix For: 3.8.0
>
>
> Code was introduced in version 3.6.0 which assumes that all JAR file URIs are
> composed of two parts - the path to the JAR file, and the path to the
> resource within the JAR file, separated by a an exclamation mark.
> This means that deployments using "uber" jars now fail if the schema is
> located inside a JAR inside BOOT-INF/lib within the top level JAR.
> A microservice deployment with a schema URI such as this fails:
> {{jar:file:/home/serviceuser/service.jar!/BOOT-INF/lib/some-other-jar.jar!/some-schema.dfdl.xsd}}
> The code at
> [https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala#L185
>
> |https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala#L185]then
> fails on the assertion that there are 2 parts after it splits by the
> separator.
> Versions up to 3.5.0 work fine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DAFFODIL-2894) Stabalize and document pluggable charset API
Steve Lawrence created DAFFODIL-2894: Summary: Stabalize and document pluggable charset API Key: DAFFODIL-2894 URL: https://issues.apache.org/jira/browse/DAFFODIL-2894 Project: Daffodil Issue Type: Bug Components: API Reporter: Steve Lawrence Fix For: 4.0.0 Daffodil supports the ability to have pluggable character sets. This is supported, but the API is not really documented and is not part of the "public" api. There aren't many known external charsets, but it's possible that might change. Like we did with Layers, we may want to take another look at our character set API and see if it is sufficient move it from an experimental API to a more public stable API. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2893) Unable to initialise
[
https://issues.apache.org/jira/browse/DAFFODIL-2893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2893.
--
Fix Version/s: 3.8.0
Resolution: Fixed
This looks like a duplicate of DAFFODIL-2892, which should be fixed by this PR:
https://github.com/apache/daffodil/pull/1223
> Unable to initialise
> -
>
> Key: DAFFODIL-2893
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2893
> Project: Daffodil
> Issue Type: Bug
> Components: DFDL Schemas, SAX
>Affects Versions: 3.6.0
>Reporter: Reshma
>Priority: Blocker
> Fix For: 3.8.0
>
>
> Because of the recent changes in Misc.scala of Release 3.6.0 to resolve jar
> path differently, I am unable to create compiler.
> The method optRelativeJarFileURI in org.apache.daffodil.lib.util.Misc.scala
> assumes to have onl two parts of the jar file location. Hence the following
> assert exists
> Assert.invariant(parts.length == 2)
> For me, i am using smooks as a dependency in my project which has daffodil
> included as a dependency. As a result i see following file path getting used
> as contextURI. When i use jar created for my project to deploy on a docker, i
> get this. The code seems to be working in my local but deploying it on docker
> makes it unfunctional.
>
> jar:file:/dpw-test-1.0.0-SNAPSHOT-exe.jar{color:#de350b}*!/*{color}BOOT-INF/lib/daffodil-lib_2.12-3.6.0.jar{color:#de350b}*!/*{color}org/apache/daffodil/xsd/DFDL_part3_model.xsd
>
>
> Because of assertion i get to see following stacktrace
> Caused by: org.smooks.api.SmooksConfigException:
> org.smooks.api.SmooksConfigException: java.lang.RuntimeException:
> org.apache.daffodil.lib.exceptions.Abort: Invariant broken: parts.length.==(2)
> org.apache.daffodil.lib.exceptions.Assert$.abort(Assert.scala:159)
> org.apache.daffodil.lib.util.Misc$.optRelativeJarFileURI(Misc.scala:183)
> org.apache.daffodil.lib.util.Misc$.getResourceRelativeOnlyOption(Misc.scala:145)
> org.apache.daffodil.lib.xml.XMLUtils$.$anonfun$resolveSchemaLocation$3(XMLUtils.scala:1492)
> at
> org.smooks.cartridges.dfdl.unparser.DfdlUnparserContentHandlerFactory.create(DfdlUnparserContentHandlerFactory.java:74)
> at
> org.smooks.cartridges.dfdl.unparser.DfdlUnparserContentHandlerFactory.create(DfdlUnparserContentHandlerFactory.java:56)
> at
> org.smooks.engine.delivery.DefaultContentDeliveryConfigBuilder$ContentHandlerExtractionStrategy.addContentDeliveryUnit(DefaultContentDeliveryConfigBuilder.java:442)
> at
> org.smooks.engine.delivery.DefaultContentDeliveryConfigBuilder$ContentHandlerExtractionStrategy.applyContentDeliveryUnitStrategy(DefaultContentDeliveryConfigBuilder.java:399)
> at
> org.smooks.engine.delivery.DefaultContentDeliveryConfigBuilder$ContentHandlerExtractionStrategy.applyStrategy(DefaultContentDeliveryConfigBuilder.java:373)
> at
> org.smooks.engine.delivery.DefaultContentDeliveryConfigBuilder$ResourceConfigTableIterator.iterate(DefaultContentDeliveryConfigBuilder.java:524)
> at
> org.smooks.engine.delivery.DefaultContentDeliveryConfigBuilder$ResourceConfigTableIterator.access$200(DefaultContentDeliveryConfigBuilder.java:504)
> at
> org.smooks.engine.delivery.DefaultContentDeliveryConfigBuilder.extractContentHandlers(DefaultContentDeliveryConfigBuilder.java:346)
> at
> org.smooks.engine.delivery.DefaultContentDeliveryConfigBuilder.load(DefaultContentDeliveryConfigBuilder.java:238)
> at
> org.smooks.engine.delivery.DefaultContentDeliveryConfigBuilder.build(DefaultContentDeliveryConfigBuilder.java:140)
> at
> org.smooks.engine.delivery.DefaultContentDeliveryRuntimeFactory.create(DefaultContentDeliveryRuntimeFactory.java:86)
> at
> org.smooks.engine.DefaultExecutionContext.(DefaultExecutionContext.java:117)
> at
> org.smooks.engine.DefaultExecutionContext.(DefaultExecutionContext.java:94)
> at org.smooks.Smooks.createExecutionContext(Smooks.java:438)
> at org.smooks.Smooks.createExecutionContext(Smooks.java:404)
> at
> org.smooks.engine.resource.visitor.smooks.NestedSmooksVisitor.filterSource(NestedSmooksVisitor.java:398)
> at
> org.smooks.engine.resource.visitor.smooks.NestedSmooksVisitor.replaceBefore(NestedSmooksVisitor.java:325)
> at
> org.smooks.engine.resource.visitor.smooks.NestedSmooksVisitor.visitBefore(NestedSmooksVisitor.java:264)
> at
> org.smooks.engine.delivery.interceptor.AbstractInterceptorVisitor$VisitBeforeInvocation.invoke(AbstractInterceptorVisitor.java:178)
> at
> org.smooks.engine.delivery.interceptor.AbstractInterceptorVisitor$VisitBeforeInvocation.invoke(AbstractInterceptorVisitor.java:175)
> at
> org.smooks.engine.delivery.interceptor.AbstractInterceptorVisitor.intercept(AbstractInterceptorVisitor.java:122)
> at
> org.smooks.engine.delivery.interceptor.EventInterceptor.visitBefore(EventInterceptor.java:67)
> at
>
[jira] [Commented] (DAFFODIL-2892) JAR file URI processing does not allow for JARs within JARs
[
https://issues.apache.org/jira/browse/DAFFODIL-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839782#comment-17839782
]
Steve Lawrence commented on DAFFODIL-2892:
--
Great! Thanks for verification!
> JAR file URI processing does not allow for JARs within JARs
> ---
>
> Key: DAFFODIL-2892
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2892
> Project: Daffodil
> Issue Type: Bug
> Components: General
>Affects Versions: 3.6.0, 3.7.0
> Environment: OpenJDK Runtime Environment Temurin-11.0.22+7 (build
> 11.0.22+7)
>Reporter: Simon Godden
>Assignee: Steve Lawrence
>Priority: Critical
> Fix For: 3.8.0
>
>
> Code was introduced in version 3.6.0 which assumes that all JAR file URIs are
> composed of two parts - the path to the JAR file, and the path to the
> resource within the JAR file, separated by a an exclamation mark.
> This means that deployments using "uber" jars now fail if the schema is
> located inside a JAR inside BOOT-INF/lib within the top level JAR.
> A microservice deployment with a schema URI such as this fails:
> {{jar:file:/home/serviceuser/service.jar!/BOOT-INF/lib/some-other-jar.jar!/some-schema.dfdl.xsd}}
> The code at
> [https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala#L185
>
> |https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala#L185]then
> fails on the assertion that there are 2 parts after it splits by the
> separator.
> Versions up to 3.5.0 work fine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DAFFODIL-2892) JAR file URI processing does not allow for JARs within JARs
[
https://issues.apache.org/jira/browse/DAFFODIL-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence updated DAFFODIL-2892:
-
Fix Version/s: 3.8.0
> JAR file URI processing does not allow for JARs within JARs
> ---
>
> Key: DAFFODIL-2892
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2892
> Project: Daffodil
> Issue Type: Bug
> Components: General
>Affects Versions: 3.6.0, 3.7.0
> Environment: OpenJDK Runtime Environment Temurin-11.0.22+7 (build
> 11.0.22+7)
>Reporter: Simon Godden
>Assignee: Steve Lawrence
>Priority: Critical
> Fix For: 3.8.0
>
>
> Code was introduced in version 3.6.0 which assumes that all JAR file URIs are
> composed of two parts - the path to the JAR file, and the path to the
> resource within the JAR file, separated by a an exclamation mark.
> This means that deployments using "uber" jars now fail if the schema is
> located inside a JAR inside BOOT-INF/lib within the top level JAR.
> A microservice deployment with a schema URI such as this fails:
> {{jar:file:/home/serviceuser/service.jar!/BOOT-INF/lib/some-other-jar.jar!/some-schema.dfdl.xsd}}
> The code at
> [https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala#L185
>
> |https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala#L185]then
> fails on the assertion that there are 2 parts after it splits by the
> separator.
> Versions up to 3.5.0 work fine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2892) JAR file URI processing does not allow for JARs within JARs
[
https://issues.apache.org/jira/browse/DAFFODIL-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17839742#comment-17839742
]
Steve Lawrence commented on DAFFODIL-2892:
--
[~sgodden], I've create [PR#1223|https://github.com/apache/daffodil/pull/1223]
to fix this, but it's hard to test since I think nested jar support is
something added by Spring Boot, which I'm not familiar with. Is it possible for
you to tested this and confirm if it resolves the issue?
> JAR file URI processing does not allow for JARs within JARs
> ---
>
> Key: DAFFODIL-2892
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2892
> Project: Daffodil
> Issue Type: Bug
> Components: General
>Affects Versions: 3.6.0, 3.7.0
> Environment: OpenJDK Runtime Environment Temurin-11.0.22+7 (build
> 11.0.22+7)
>Reporter: Simon Godden
>Assignee: Steve Lawrence
>Priority: Critical
>
> Code was introduced in version 3.6.0 which assumes that all JAR file URIs are
> composed of two parts - the path to the JAR file, and the path to the
> resource within the JAR file, separated by a an exclamation mark.
> This means that deployments using "uber" jars now fail if the schema is
> located inside a JAR inside BOOT-INF/lib within the top level JAR.
> A microservice deployment with a schema URI such as this fails:
> {{jar:file:/home/serviceuser/service.jar!/BOOT-INF/lib/some-other-jar.jar!/some-schema.dfdl.xsd}}
> The code at
> [https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala#L185
>
> |https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala#L185]then
> fails on the assertion that there are 2 parts after it splits by the
> separator.
> Versions up to 3.5.0 work fine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (DAFFODIL-2892) JAR file URI processing does not allow for JARs within JARs
[
https://issues.apache.org/jira/browse/DAFFODIL-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence reassigned DAFFODIL-2892:
Assignee: Steve Lawrence
> JAR file URI processing does not allow for JARs within JARs
> ---
>
> Key: DAFFODIL-2892
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2892
> Project: Daffodil
> Issue Type: Bug
> Components: General
>Affects Versions: 3.6.0, 3.7.0
> Environment: OpenJDK Runtime Environment Temurin-11.0.22+7 (build
> 11.0.22+7)
>Reporter: Simon Godden
>Assignee: Steve Lawrence
>Priority: Critical
>
> Code was introduced in version 3.6.0 which assumes that all JAR file URIs are
> composed of two parts - the path to the JAR file, and the path to the
> resource within the JAR file, separated by a an exclamation mark.
> This means that deployments using "uber" jars now fail if the schema is
> located inside a JAR inside BOOT-INF/lib within the top level JAR.
> A microservice deployment with a schema URI such as this fails:
> {{jar:file:/home/serviceuser/service.jar!/BOOT-INF/lib/some-other-jar.jar!/some-schema.dfdl.xsd}}
> The code at
> [https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala#L185
>
> |https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/util/Misc.scala#L185]then
> fails on the assertion that there are 2 parts after it splits by the
> separator.
> Versions up to 3.5.0 work fine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2891) Abort: Invariant broken: minReps.==(0L) when unparsing
[
https://issues.apache.org/jira/browse/DAFFODIL-2891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2891.
--
Resolution: Fixed
Fixed in commit 5ca4319c83da3d5caf5d9e17ce6a05a021e621ab
> Abort: Invariant broken: minReps.==(0L) when unparsing
> --
>
> Key: DAFFODIL-2891
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2891
> Project: Daffodil
> Issue Type: Bug
> Components: Unparsing
>Reporter: Steve Lawrence
>Assignee: Steve Lawrence
>Priority: Major
> Fix For: 3.8.0
>
>
> If we have a schema with an array with occursCountKind="expression" and
> minOccurs greater than zero, and we unparse an infoset where there are zero
> occurrences, we get an abort assertion. This is not a valid infoset, but we
> should get an UnparseError instead of an abort.
> *Schema:*
> {code:xml}
>
>
>
>
>
>
>
>dfdl:occursCountKind="expression" dfdl:occursCount="{
> ../field1Occurs }" />
>dfdl:occursCountKind="parsed" />
>
>
>
> {code}
> *Infoset:*
> {code:xml}
> http://example.com;>
> 0
> 1
> 2
> 3
> 4
>
> {code}
> *Error:*
> {code}
> org.apache.daffodil.lib.exceptions.Abort: Invariant broken: minReps.==(0L)
> org.apache.daffodil.lib.exceptions.Assert$.abort(Assert.scala:159)
> org.apache.daffodil.unparsers.runtime1.OrderedUnseparatedSequenceUnparser.unparse(UnseparatedSequenceUnparsers.scala:159)
> org.apache.daffodil.runtime1.processors.unparsers.Unparser.unparse1(Unparser.scala:75)
> org.apache.daffodil.runtime1.processors.unparsers.Unparser.unparse1$(Unparser.scala:38)
> at org.apache.daffodil.lib.exceptions.Assert$.abort(Assert.scala:159)
> at
> org.apache.daffodil.unparsers.runtime1.OrderedUnseparatedSequenceUnparser.unparse(UnseparatedSequenceUnparsers.scala:159)
> at
> org.apache.daffodil.runtime1.processors.unparsers.Unparser.unparse1(Unparser.scala:75)
> at
> org.apache.daffodil.runtime1.processors.unparsers.Unparser.unparse1$(Unparser.scala:38)
> at
> org.apache.daffodil.runtime1.processors.unparsers.CombinatorUnparser.unparse1(Unparser.scala:124)
> at
> org.apache.daffodil.unparsers.runtime1.ElementUnparserBase.runContentUnparser(ElementUnparser.scala:181)
> at
> org.apache.daffodil.unparsers.runtime1.ElementUnparserBase.unparse(ElementUnparser.scala:202)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DAFFODIL-2891) Abort: Invariant broken: minReps.==(0L) when unparsing
Steve Lawrence created DAFFODIL-2891:
Summary: Abort: Invariant broken: minReps.==(0L) when unparsing
Key: DAFFODIL-2891
URL: https://issues.apache.org/jira/browse/DAFFODIL-2891
Project: Daffodil
Issue Type: Bug
Components: Unparsing
Reporter: Steve Lawrence
Assignee: Steve Lawrence
Fix For: 3.8.0
If we have a schema with an array with occursCountKind="expression" and
minOccurs greater than zero, and we unparse an infoset where there are zero
occurrences, we get an abort assertion. This is not a valid infoset, but we
should get an UnparseError instead of an abort.
*Schema:*
{code:xml}
{code}
*Infoset:*
{code:xml}
http://example.com;>
0
1
2
3
4
{code}
*Error:*
{code}
org.apache.daffodil.lib.exceptions.Abort: Invariant broken: minReps.==(0L)
org.apache.daffodil.lib.exceptions.Assert$.abort(Assert.scala:159)
org.apache.daffodil.unparsers.runtime1.OrderedUnseparatedSequenceUnparser.unparse(UnseparatedSequenceUnparsers.scala:159)
org.apache.daffodil.runtime1.processors.unparsers.Unparser.unparse1(Unparser.scala:75)
org.apache.daffodil.runtime1.processors.unparsers.Unparser.unparse1$(Unparser.scala:38)
at org.apache.daffodil.lib.exceptions.Assert$.abort(Assert.scala:159)
at
org.apache.daffodil.unparsers.runtime1.OrderedUnseparatedSequenceUnparser.unparse(UnseparatedSequenceUnparsers.scala:159)
at
org.apache.daffodil.runtime1.processors.unparsers.Unparser.unparse1(Unparser.scala:75)
at
org.apache.daffodil.runtime1.processors.unparsers.Unparser.unparse1$(Unparser.scala:38)
at
org.apache.daffodil.runtime1.processors.unparsers.CombinatorUnparser.unparse1(Unparser.scala:124)
at
org.apache.daffodil.unparsers.runtime1.ElementUnparserBase.runContentUnparser(ElementUnparser.scala:181)
at
org.apache.daffodil.unparsers.runtime1.ElementUnparserBase.unparse(ElementUnparser.scala:202)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2890) Set SOURCE_DATE_EPOCH when creating releases
[ https://issues.apache.org/jira/browse/DAFFODIL-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2890. -- Resolution: Fixed Fixed in commit 65b68e5dc17a80be7877d8b44e63c9353cb8ab86 > Set SOURCE_DATE_EPOCH when creating releases > > > Key: DAFFODIL-2890 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2890 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.8.0 > > > The > [SOURCE_DATE_EPOCH|https://reproducible-builds.org/docs/source-date-epoch/] > environment variable is one way to create reproducible builds. The latest > release of sbt-native-packager adds support for this environment variable, we > should set this in our release candidate script so that the artifacts we > generate are reproducible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2890) Set SOURCE_DATE_EPOCH when creating releases
Steve Lawrence created DAFFODIL-2890: Summary: Set SOURCE_DATE_EPOCH when creating releases Key: DAFFODIL-2890 URL: https://issues.apache.org/jira/browse/DAFFODIL-2890 Project: Daffodil Issue Type: Bug Components: Infrastructure Reporter: Steve Lawrence Assignee: Steve Lawrence Fix For: 3.8.0 The [SOURCE_DATE_EPOCH|https://reproducible-builds.org/docs/source-date-epoch/] environment variable is one way to create reproducible builds. The latest release of sbt-native-packager adds support for this environment variable, we should set this in our release candidate script so that the artifacts we generate are reproducible. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2889) Intermitted CI failures missing mxml.h on macOS
[
https://issues.apache.org/jira/browse/DAFFODIL-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17836153#comment-17836153
]
Steve Lawrence commented on DAFFODIL-2889:
--
CI was pinned to 3.3.1 in commit 243612f182b18bc1e6565b5452c36c17fb79f34b
I think I'll leave this open since we probably want to consider this short-term
solution.
Long term, we probably want to figure out why 4.x doesn't work.
> Intermitted CI failures missing mxml.h on macOS
> ---
>
> Key: DAFFODIL-2889
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2889
> Project: Daffodil
> Issue Type: Bug
>Reporter: Steve Lawrence
>Priority: Major
>
> I've noticed a handful of intermittent CI failures over the past couple of
> weeks with the macOS build relating to a missing mxml header file. The error
> during compilation is:
> {quote}
> [info] [cc, -c,
> /Users/runner/work/daffodil/daffodil/daffodil-codegen-c/src/main/resources/org/apache/daffodil/codegen/c/files/libcli/xml_writer.c,
> -o,
> /Users/runner/work/daffodil/daffodil/daffodil-codegen-c/target/streams/compile/ccSourceObjectMap/_global/streams/compile/libruntime.a/src/main/resources/org/apache/daffodil/codegen/c/files/libcli/xml_writer.c.o,
>
> -I/Users/runner/work/daffodil/daffodil/daffodil-codegen-c/src/main/resources/org/apache/daffodil/codegen/c/files/libcli,
>
> -I/Users/runner/work/daffodil/daffodil/daffodil-codegen-c/src/main/resources/org/apache/daffodil/codegen/c/files/libruntime,
> -Wall, -Wextra, -Wpedantic, -std=gnu11]
> In file included from
> /Users/runner/work/daffodil/daffodil/daffodil-codegen-c/src/main/resources/org/apache/daffodil/codegen/c/files/libcli/xml_writer.c:20:
> In file included from
> /Users/runner/work/daffodil/daffodil/daffodil-codegen-c/src/main/resources/org/apache/daffodil/codegen/c/files/libcli/xml_writer.h:25:
> /Users/runner/work/daffodil/daffodil/daffodil-codegen-c/src/main/resources/org/apache/daffodil/codegen/c/files/libcli/stack.h:23:10:
> fatal error: 'mxml.h' file not found
> #include // for mxml_node_t
> ^~~~
> 1 error generated.
> {quote}
> Note that on the same build installing mxml seems to work fine in the Install
> Dependencies (macOS) step:
> {quote}
> Run brew install libmxml
> ==> Downloading https://ghcr.io/v2/homebrew/core/libmxml/manifests/4.0.2
> ==> Fetching libmxml
> ==> Downloading
> https://ghcr.io/v2/homebrew/core/libmxml/blobs/sha256:98e1e1b8b65b2cabd74cd88dc9c6bc7c20b702729047bc80bb90a692ae8d4357
> ==> Pouring libmxml--4.0.2.monterey.bottle.tar.gz
> /usr/local/Cellar/libmxml/4.0.2: 19 files, 1.8MB
> {quote}
> Usually rerunning the job fixes the issue, but it would be nice to figure out
> what is going so we can avoid the random failures. Unfortunately, it's
> difficult to reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DAFFODIL-2889) Intermitted CI failures missing mxml.h on macOS
Steve Lawrence created DAFFODIL-2889:
Summary: Intermitted CI failures missing mxml.h on macOS
Key: DAFFODIL-2889
URL: https://issues.apache.org/jira/browse/DAFFODIL-2889
Project: Daffodil
Issue Type: Bug
Reporter: Steve Lawrence
I've noticed a handful of intermittent CI failures over the past couple of
weeks with the macOS build relating to a missing mxml header file. The error
during compilation is:
{quote}
[info] [cc, -c,
/Users/runner/work/daffodil/daffodil/daffodil-codegen-c/src/main/resources/org/apache/daffodil/codegen/c/files/libcli/xml_writer.c,
-o,
/Users/runner/work/daffodil/daffodil/daffodil-codegen-c/target/streams/compile/ccSourceObjectMap/_global/streams/compile/libruntime.a/src/main/resources/org/apache/daffodil/codegen/c/files/libcli/xml_writer.c.o,
-I/Users/runner/work/daffodil/daffodil/daffodil-codegen-c/src/main/resources/org/apache/daffodil/codegen/c/files/libcli,
-I/Users/runner/work/daffodil/daffodil/daffodil-codegen-c/src/main/resources/org/apache/daffodil/codegen/c/files/libruntime,
-Wall, -Wextra, -Wpedantic, -std=gnu11]
In file included from
/Users/runner/work/daffodil/daffodil/daffodil-codegen-c/src/main/resources/org/apache/daffodil/codegen/c/files/libcli/xml_writer.c:20:
In file included from
/Users/runner/work/daffodil/daffodil/daffodil-codegen-c/src/main/resources/org/apache/daffodil/codegen/c/files/libcli/xml_writer.h:25:
/Users/runner/work/daffodil/daffodil/daffodil-codegen-c/src/main/resources/org/apache/daffodil/codegen/c/files/libcli/stack.h:23:10:
fatal error: 'mxml.h' file not found
#include // for mxml_node_t
^~~~
1 error generated.
{quote}
Note that on the same build installing mxml seems to work fine in the Install
Dependencies (macOS) step:
{quote}
Run brew install libmxml
==> Downloading https://ghcr.io/v2/homebrew/core/libmxml/manifests/4.0.2
==> Fetching libmxml
==> Downloading
https://ghcr.io/v2/homebrew/core/libmxml/blobs/sha256:98e1e1b8b65b2cabd74cd88dc9c6bc7c20b702729047bc80bb90a692ae8d4357
==> Pouring libmxml--4.0.2.monterey.bottle.tar.gz
/usr/local/Cellar/libmxml/4.0.2: 19 files, 1.8MB
{quote}
Usually rerunning the job fixes the issue, but it would be nice to figure out
what is going so we can avoid the random failures. Unfortunately, it's
difficult to reproduce.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DAFFODIL-2888) Incorrect out of range warning with constant dfdl:length and length/minLength/maxLength facets
Steve Lawrence created DAFFODIL-2888:
Summary: Incorrect out of range warning with constant dfdl:length
and length/minLength/maxLength facets
Key: DAFFODIL-2888
URL: https://issues.apache.org/jira/browse/DAFFODIL-2888
Project: Daffodil
Issue Type: Bug
Components: Front End
Reporter: Steve Lawrence
Say we have a schema like this:
{code:xml}
{code}
So we have a 2-byte constant length string and we specify maxLength should be
2. However, because we specify the length in bits, the value of dfdl:length is
greater than the value of maxLength and so we output a warning:
{quote}
[warn] Schema Definition Warning: Explicit dfdl:length of 16 is out of range
for facet maxLength='2'. (id: facetExplicitLengthOutOfRange)
{quote}
The issue is this check does not take into account the units of length. It also
does not take into account the character set of the data. For example, 1-byte
of the X-DFDL-BITS-MSBF character set has a minLength of 8 characters in the
infoset.
There are other things to consider, for example, maybe we have a string with
dfdl:length="3" with units of bytes, but we always expect to have 1 character
of padding removed, so our maxLength wants to be 2. We would get a warning here
too.
Maybe some false positives are okay, but we can probably improve the logic to
avoid false positives in some cases at least to take into account units and
charsets? Or maybe this warning isn't really that helpful, and the compelxity
needed to get it right is not worth it, and should be removed?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2885) Release Daffodil 3.7.0
[ https://issues.apache.org/jira/browse/DAFFODIL-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2885. -- Resolution: Fixed Prepared in commit fbf5343ba888580fffcd17b310244aa8a5d1d1ab. Tagged as v3.7.0 and released. > Release Daffodil 3.7.0 > -- > > Key: DAFFODIL-2885 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2885 > Project: Daffodil > Issue Type: Bug >Reporter: Peter Katlic >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.7.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2887) Choice branch array with discriminator or direct dispatch broken
Steve Lawrence created DAFFODIL-2887:
Summary: Choice branch array with discriminator or direct dispatch
broken
Key: DAFFODIL-2887
URL: https://issues.apache.org/jira/browse/DAFFODIL-2887
Project: Daffodil
Issue Type: Bug
Components: Front End
Affects Versions: 3.7.0
Reporter: Steve Lawrence
Say we have this schema:
{code:xml}
http://www.w3.org/2001/XMLSchema;
xmlns:xs="http://www.w3.org/2001/XMLSchema;
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/;
xmlns:ex="http://example.com;
targetNamespace="http://example.com;
elementFormDefault="unqualified">
http://www.ogf.org/dfdl/;>
http://www.ogf.org/dfdl/;>
http://www.ogf.org/dfdl/;>
{code}
So it is a space separated list of items, where the first is either "foo" or
"bar", and that defines the type of all following items to be an array of Foo's
or an array of Bar's.
This currently fails to compile with the error
{code}
Relative path '../../Type eq foo' past root element
{code}
If we remove an up step and change the discriminators to "../Type eq 'foo'",
then we get the error:
{code}
No element corresponding to step {}Type found. Possibilities for this step
include: {}Foo, {}Bar.
{code}
So there is no way to reference Type using up-steps.
A workaround is to wrap the array elemens in a sequence so the choice branches
are now sequenes instead of arrays. Doing this allows the scheme to work. But
when a choice branch is an array, there is an implied surrounding sequence so
these should be exactly the same.
Potentially related, if we change the above snippet to remove the
discriminators and instead use direct dispatch and put dfdl:choiceDispatchKey
on the choice and dfdl:choiceBranchKey on the Foo/Bar arrays, we get the
warning and error:
{code}
DFDL property was ignored: choiceBranchKey="foo" (id: ignoreDFDLProperty)
[error] Parse Error: Choice dispatch branch failed: List(Parse Error: Array
element parsed succesfully, but consumed no data and is stuck in an infinite
loop as it is unbounded.
{code}
Something seems to be wrong with our implied sequence choice branches.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (DAFFODIL-2885) Release Daffodil 3.7.0
[ https://issues.apache.org/jira/browse/DAFFODIL-2885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence reassigned DAFFODIL-2885: Assignee: Steve Lawrence (was: Peter Katlic) > Release Daffodil 3.7.0 > -- > > Key: DAFFODIL-2885 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2885 > Project: Daffodil > Issue Type: Bug >Reporter: Peter Katlic >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.7.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2243) Runner runAllTest method
[ https://issues.apache.org/jira/browse/DAFFODIL-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2243. -- Fix Version/s: 3.7.0 Resolution: Fixed > Runner runAllTest method > > > Key: DAFFODIL-2243 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2243 > Project: Daffodil > Issue Type: Improvement > Components: TDML Runner >Affects Versions: 2.4.0 >Reporter: Alexander Deutschmann >Priority: Minor > Labels: beginner > Fix For: 3.7.0 > > > Hello, > can you make a runAllTest() method available? In Version 2.4.0 i cant find > any method. > Or is it possible to read all testcases from the Runner class? > I found this in the DfdlTestSuite Class. > Thank you :) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2243) Runner runAllTest method
[ https://issues.apache.org/jira/browse/DAFFODIL-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832826#comment-17832826 ] Steve Lawrence commented on DAFFODIL-2243: -- Yep, this does already exist, and those tests verify it. Note that commit d331578714ad8ef2f8147a3ce1c3cbac1980effb only added runAllTests to the DFDLTestSuite. I think at that time of that commit we did not have the "Runner" class yet. At some point we added the Runner class (which is what users should use now) but it was only able to run a single test--it had no runAllTests function. I'm not sure when that was, but I can confirm it still did not have runAllTests for v2.4.0 when this ticket was created, and probably why this ticket was opened. But it looks like commit [0700ee8dc9531497f3e8b0fdf9266f8e3b105c27|https://github.com/apache/daffodil/commit/0700ee8dc9531497f3e8b0fdf9266f8e3b105c27] added a runAllTests to the Runner. That commit was part of the 3.2.0 release, so this has been fixed for a while. This can be resolved. > Runner runAllTest method > > > Key: DAFFODIL-2243 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2243 > Project: Daffodil > Issue Type: Improvement > Components: TDML Runner >Affects Versions: 2.4.0 >Reporter: Alexander Deutschmann >Priority: Minor > Labels: beginner > > Hello, > can you make a runAllTest() method available? In Version 2.4.0 i cant find > any method. > Or is it possible to read all testcases from the Runner class? > I found this in the DfdlTestSuite Class. > Thank you :) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DAFFODIL-2886) Implement HexBinary Comparison Operations
[ https://issues.apache.org/jira/browse/DAFFODIL-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence updated DAFFODIL-2886: - Issue Type: New Feature (was: Bug) > Implement HexBinary Comparison Operations > - > > Key: DAFFODIL-2886 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2886 > Project: Daffodil > Issue Type: New Feature > Components: Back End >Affects Versions: 3.6.0 >Reporter: Olabusayo Kilo >Priority: Minor > Fix For: 4.0.0 > > > Equality and Non Equality are already implemented. But LT, LE, GT and GE > needs to be implemented for hexBinary. The Comparison classes exist, but > result in a Assert.usageError if operate is called on either of those 4 > comparisons. > If we wait till we drop support for Java 8 we can use > [Arrays.compare()|https://docs.oracle.com/javase%2F9%2Fdocs%2Fapi%2F%2F/java/util/Arrays.html#compare-byte:A-byte:A-] > so it should be a trivial change. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DAFFODIL-2886) Implement HexBinary Comparison Operations
[ https://issues.apache.org/jira/browse/DAFFODIL-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence updated DAFFODIL-2886: - Fix Version/s: 4.0.0 (was: 3.7.0) > Implement HexBinary Comparison Operations > - > > Key: DAFFODIL-2886 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2886 > Project: Daffodil > Issue Type: Bug > Components: Back End >Affects Versions: 3.6.0 >Reporter: Olabusayo Kilo >Priority: Minor > Fix For: 4.0.0 > > > Equality and Non Equality are already implemented. But LT, LE, GT and GE > needs to be implemented for hexBinary. The Comparison classes exist, but > result in a Assert.usageError if operate is called on either of those 4 > comparisons. > If we wait till we drop support for Java 8 we can use > [Arrays.compare()|https://docs.oracle.com/javase%2F9%2Fdocs%2Fapi%2F%2F/java/util/Arrays.html#compare-byte:A-byte:A-] > so it should be a trivial change. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2688) prefixed length field when not enough bits available gives Runtime SDE not parse error
[
https://issues.apache.org/jira/browse/DAFFODIL-2688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2688.
--
Fix Version/s: 3.7.0
Resolution: Fixed
Fixed in commit 4c61dc53bacc4dc6fb416512d8cef8b8a8a2c9bb
> prefixed length field when not enough bits available gives Runtime SDE not
> parse error
> --
>
> Key: DAFFODIL-2688
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2688
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 3.3.0
>Reporter: Mike Beckerle
>Assignee: Mike Beckerle
>Priority: Major
> Fix For: 3.7.0
>
>
> When a prefixed length, let's say a 4-byte unsigned int, is being parsed, and
> there are not 32-bits available (because, for example you are at end of
> data), then you don't get a parse error. Instead you get:
> {code:java}
> Runtime Schema Definition Error: Prefixed length result must be non-negative
> after dfdl:prefixIncludesPrefixLength adjustment , but was: -4 {code}
> This needs to be a parse error due to the insufficient bits.
> If you trace, you get this:
> {code:java}
> parser:
> bitPosition: 1920
> data:
> │ │
> 87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
> infoset:
>
>
> diff:
> (no differences)
> failure:
> Parse Error: Insufficient bits in data. Needed 32 bit(s) but found only 0
> available.
> Schema context: message (prefixLength) Location line 385 in
> file:/../common.dfdl.xsd
> Data location was preceding byte 240 limit(bytes) 240
> -
> 347org.apache.daffodil.tdml.TDMLExceptionImpl: (Implementation: daffodil)
> Runtime Schema Definition Error: Prefixed length result must be non-negative
> after dfdl:prefixIncludesPrefixLength adjustment , but was: -4 {code}
> So you can see how it's just out of data, yet getting an RSDE on this.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2884) String-As-XML cause SDE on malformed XML data. Needs to be PE.
[ https://issues.apache.org/jira/browse/DAFFODIL-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831427#comment-17831427 ] Steve Lawrence commented on DAFFODIL-2884: -- It feels like maybe there are two similar but orthogonal concepts to here: 1) Is parsed/unparsed data valid XML? 2) How should that XML string data be projected into and out of infoset? For example, we might be projecting into a JSON infoset where xml data just becomes an JSON string, but we do still want to know if it is valid to allow allow for backtracking. And the reverse, one could imagine having JSON in our data straem and we want to ensure it's valid json, but still project it to an XML infoset where it is just a normal string. So it feels like we don't really want to change our stringAsXml runtime property implementation for the XML Text infoset inputtter/outputter, since that handles concept 2. But we do need a new concept to allow validating/normalizing xs:string data. On option could be new DPath functions, e.g. dfdlx:isXML(xs:string) or dfdlx:isJSON(xs:string). These functions could be used in a discriminator to backtrack if the parsed string isn't valid XML. These could also be implemented as UDFs in current versions of Daffodil, but is probably something we should include. Another option is maybe a new property, e.g. dfdlx:textStringRep, which could have values of "standard" (i.e. current behavior), "xml", "json", or maybe more. This would do what you describe and validate/normalize the parsed string value according to the representation. The infoset value would still be a string, and infoset outputters/inputters would still need logic to determine how best to project them into the infoset using runtimeProperties. Any other approaches? > String-As-XML cause SDE on malformed XML data. Needs to be PE. > -- > > Key: DAFFODIL-2884 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2884 > Project: Daffodil > Issue Type: Bug > Components: Back End >Affects Versions: 3.6.0 >Reporter: Mike Beckerle >Priority: Major > > When using the string-as-XML feature, currently if the string that is > supposed to be XML is malformed, then a WstxUnexpectedCharException (or other > similar exception) gets thrown in the InfosetOutputter which is what does the > string-of-XML to actual XML conversion. The InfosetOutputter is outside the > scope of backtracking, so this error cannot be converted into a ParseError at > this point. The InfosetOutputter currently escalates this to an SDE. > That's not correct for a data problem. The parser could be speculating down a > path where the string of data that is supposed to be XML is just gibberish. > If that string is malformed XML, a Parse Error needs to occur so we can > backtrack. > Converting Infoset into XML is normally something done by the > InfosetOutputter, but in this case it cannot be. It needs to be done in the > string parser, and the Infoset needs to somehow cache the resulting XML so it > can be handed off to the InfosetOutputter. > I think this has to work analogously to text numbers. We parse the string > first, then convert to the data type, which for numbers is an > integer/float/decimal, etc. This conversion can fail, and that's a Parse > Error. String-as-XML needs to work the same way. The string is parsed via one > of the lengthKind techniques, then it is converted into XML. If the > conversion to XML fails, then it's a Parse Error. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2880) CLI tunable option does not have an affect using saved parser -- needs error/warning
[ https://issues.apache.org/jira/browse/DAFFODIL-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2880. -- Fix Version/s: 3.7.0 Resolution: Fixed Fixed in commit 9b12912f59140b1e934133e777a616b6f8fff702 > CLI tunable option does not have an affect using saved parser -- needs > error/warning > > > Key: DAFFODIL-2880 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2880 > Project: Daffodil > Issue Type: Bug > Components: CLI >Reporter: Steve Lawrence >Priority: Minor > Fix For: 3.7.0 > > > When using the CLI with the --parser option to use a saved parser, the -T > option to set tunables has no affect. Instead, Daffodil uses the values of > tunables set at schema compilation time. > This is probably the correct behavior since it could potentially cause odd > behavior to have different tunables for schema compilation and schema > parsing, but there is no warning or error message that they are ignored and > have no affect. So Daffodil just silently ignores the tunable values, which > can be confusing if you are expecting a different behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2877) Allow pattern restriction facet to be used on non-string types
[
https://issues.apache.org/jira/browse/DAFFODIL-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2877.
--
Resolution: Fixed
Fixed in commit 77d61900715034c9b09c73e93e9bedfc16d81438
> Allow pattern restriction facet to be used on non-string types
> --
>
> Key: DAFFODIL-2877
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2877
> Project: Daffodil
> Issue Type: New Feature
> Components: Back End, Front End
>Reporter: Steve Lawrence
>Priority: Major
>
> Section 5.2 of the DFDL specification says that the pattern restriction facet
> can only be used for xs:string types. Daffodil currently implements this
> limitation.
> However, there are a number of cases where a pattern is the only way to
> correctly restrict the value of non-string types, such as xs:long.
> One example is a number that can only be between 100 and 300 but in multiples
> of 5. In that case, a pattern like this would work:
> {code}
>
> {code}
> Another example could be apartment rooms, where the first digit represents
> the floor (e.g. 1-5), and the second two digits represent the room number
> between 1 and 15 (assuming 15 rooms per floor). In that case, the pattern
> restriction might be:
> {code}
>
> {code}
> The patterns are often going to be a bit ugly, but in some cases it's the
> only way to correctly validate numbers using only XML Schema capabilities.
> Note that only supporting pattern on xs:string types is a limitation imposed
> by DFDL--XML schema allows pattern restriction on all types. We should remove
> this limitation and allow pattern facet to be used on all types.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2671) support minInclusive maxInclusive on date/time/dateTime
[
https://issues.apache.org/jira/browse/DAFFODIL-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828051#comment-17828051
]
Steve Lawrence commented on DAFFODIL-2671:
--
Actually, DFDL allows minInclusive/maxExclusiv/etc to be define for calendar
types, so the invalidRestrictionPolicy does not come into play for this
particular bug. This bug is to just implement something that DFDL says we
should support. We might still need to come up with an answer for how to handle
DFDL-invalid restrictions where we don't support the "validate" policy, but
it's not needed for this bug.
> support minInclusive maxInclusive on date/time/dateTime
> ---
>
> Key: DAFFODIL-2671
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2671
> Project: Daffodil
> Issue Type: New Feature
> Components: Back End
>Affects Versions: 3.2.1
>Reporter: Mike Beckerle
>Assignee: Mike Beckerle
>Priority: Minor
>
> I am trying to use this simple type:
> {code:java}
>
>
>
>
>
> {code}
> I get this diagnostic message:
> {code:java}
> Schema Definition Error: Failed to parse (1995-01-01T00:00:00) to dateTime
> (-MM-dd'T'HH:mm:ss.SSxxx) due to Character array is missing "e"
> notation exponential mark. (after Unparseable date:
> "1995-01-01T00:00:00").{code}
> The other facets (minExclusive, maxExclusive), and the other calendar types
> (date, time) should also be tested.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2671) support minInclusive maxInclusive on date/time/dateTime
[
https://issues.apache.org/jira/browse/DAFFODIL-2671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828049#comment-17828049
]
Steve Lawrence commented on DAFFODIL-2671:
--
DAFFODIL-2877 adds "error", "ignore", and "validate" options for how to handle
DFDL-invalid restrictions. "error" and "ignore" are pretty trivial to implement
in this case, but "validate" might be hard for date/time min/maxInclusive
because the current implementation is tied pretty heavily to numbers. If it's
too difficult to implement validation for a specific restriction (like maybe
this one) how should we handle it? Maybe we warn that the
invalidRestrictionPolicy says to "validate", but we can't validate this
specific restriction?
> support minInclusive maxInclusive on date/time/dateTime
> ---
>
> Key: DAFFODIL-2671
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2671
> Project: Daffodil
> Issue Type: New Feature
> Components: Back End
>Affects Versions: 3.2.1
>Reporter: Mike Beckerle
>Assignee: Mike Beckerle
>Priority: Minor
>
> I am trying to use this simple type:
> {code:java}
>
>
>
>
>
> {code}
> I get this diagnostic message:
> {code:java}
> Schema Definition Error: Failed to parse (1995-01-01T00:00:00) to dateTime
> (-MM-dd'T'HH:mm:ss.SSxxx) due to Character array is missing "e"
> notation exponential mark. (after Unparseable date:
> "1995-01-01T00:00:00").{code}
> The other facets (minExclusive, maxExclusive), and the other calendar types
> (date, time) should also be tested.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2879) Group evaluation throwing ClassCastException for optional element
[
https://issues.apache.org/jira/browse/DAFFODIL-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2879.
--
Fix Version/s: 3.7.0
Resolution: Fixed
Fixed in commit 3f897fe62b7946870290d7414ee27d5b733426b1
> Group evaluation throwing ClassCastException for optional element
> -
>
> Key: DAFFODIL-2879
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2879
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 3.6.0
>Reporter: Peter Katlic
>Priority: Major
> Fix For: 3.7.0
>
>
> A group is defined with an element using fn:count and referenced inside a
> sequence of array and optional elements. See the test count_21b in
> daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml
> for an example.
> Group schema:
> {code:java}
>
>
> dfdl:inputValueCalc="{ fn:count(../ex:foo) }"/>
>
> {code}
> Sequence schema:
> {code:java}
>
>
>
>
>
>
> dfdl:representation="binary"/>
>
>
>
>
>
>
>
> dfdl:representation="binary"/>
>
>
> dfdl:representation="binary" dfdl:inputValueCalc="{ fn:count(../ex:foo) }"/>
>
>
>
>
>
> {code}
> Uncommenting
> {code:java}
> {code}
> and commenting out
> {code:java}
> dfdl:inputValueCalc="{ fn:count(../ex:foo) }"/>{code}
> will produce the following error:
> {code:java}
> java.lang.ClassCastException: class
> org.apache.daffodil.runtime1.infoset.DISimple cannot be cast to class
> org.apache.daffodil.runtime1.infoset.DIArray{code}
> It seems as if the optional is evaluated as a DownArray instead of a
> DownElement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2882) Incorrect mapping from separatorPoliciy to separatorSuppressionPolicy
[
https://issues.apache.org/jira/browse/DAFFODIL-2882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2882.
--
Fix Version/s: 3.7.0
Resolution: Fixed
Fixed in commit 6edf5304a355991dba169b163f2f191a981a1f01
> Incorrect mapping from separatorPoliciy to separatorSuppressionPolicy
> -
>
> Key: DAFFODIL-2882
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2882
> Project: Daffodil
> Issue Type: Bug
> Components: Front End
>Affects Versions: 3.6.0
>Reporter: Steve Lawrence
>Priority: Major
> Fix For: 3.7.0
>
>
> Daffodil allows use of the deprecated separatorPolicy property and internally
> maps it to the new separatorSupressionPolicy property. This is done here:
> https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/schema/annotation/props/ByHandMixins.scala#L198-L201
> So our mappings are:
> {code}
> required -> never
> suppressed -> anyEmpty
> suppressedAtEndStrict -> trailingEmpty
> suppressedAtEndLax -> trailingEmptyStrict
> {code}
> However, according to the errata document
> (https://ogf.org/documents/GFD.214.pdf), the mappings should be:
> {code}
> required -> never
> suppressed -> anyEmpty
> suppressedAtEndLax -> trailingEmpty
> suppressedAtEndStrict -> trailingEmptyStrict
> {code}
> Note that suppressedAtEndStrict and suppressedAtEndLax are swapped and do not
> match the errata document. Comparing the wording of older versions of the
> DFDL spec, it does seem like our mapping is incorrect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2013) Predicate required for when accessing optional elements in expression
[ https://issues.apache.org/jira/browse/DAFFODIL-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2013. -- Resolution: Invalid This bug is wrong. Predicates are not required to access optional elements, in fact they are not allowed. Predicates are only allowed on arrays (i.e. maxOccurs > 1). Commit 36b98fc2d78b0485af614202b4cd0c51cec9a2e3 adds a test showing this is the case. Closing as "Invalid. > Predicate required for when accessing optional elements in expression > - > > Key: DAFFODIL-2013 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2013 > Project: Daffodil > Issue Type: Bug > Components: Front End >Affects Versions: 2.2.0 >Reporter: Steve Lawrence >Priority: Minor > > Behind the scenes we treat optional elements (e.g. minOccurs="1" > dfdl:lengthKind="implicit"), as arrays that should just always have either > zero or one elements. This means that when an expression attempts to access > this element, a predicate of [1] is required. Otherwise you get an error > about "Query-style paths not supported". We should have logic to treat > optional elements special so that the predicate is not mandatory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2882) Incorrect mapping from separatorPoliciy to separatorSuppressionPolicy
Steve Lawrence created DAFFODIL-2882:
Summary: Incorrect mapping from separatorPoliciy to
separatorSuppressionPolicy
Key: DAFFODIL-2882
URL: https://issues.apache.org/jira/browse/DAFFODIL-2882
Project: Daffodil
Issue Type: Bug
Components: Front End
Affects Versions: 3.6.0
Reporter: Steve Lawrence
Daffodil allows use of the deprecated separatorPolicy property and internally
maps it to the new separatorSupressionPolicy property. This is done here:
https://github.com/apache/daffodil/blob/main/daffodil-lib/src/main/scala/org/apache/daffodil/lib/schema/annotation/props/ByHandMixins.scala#L198-L201
So our mappings are:
{code}
required -> never
suppressed -> anyEmpty
suppressedAtEndStrict -> trailingEmpty
suppressedAtEndLax -> trailingEmptyStrict
{code}
However, according to the errata document
(https://ogf.org/documents/GFD.214.pdf), the mappings should be:
{code}
required -> never
suppressed -> anyEmpty
suppressedAtEndLax -> trailingEmpty
suppressedAtEndStrict -> trailingEmptyStrict
{code}
Note that suppressedAtEndStrict and suppressedAtEndLax are swapped and do not
match the errata document. Comparing the wording of older versions of the DFDL
spec, it does seem like our mapping is incorrect.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2879) Group evaluation throwing ClassCastException for optional element
[
https://issues.apache.org/jira/browse/DAFFODIL-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825731#comment-17825731
]
Steve Lawrence commented on DAFFODIL-2879:
--
> Though, I thought we had an issue where optionals required a predicate? So
> maybe scalars and optionals can't be allowed either? So maybe a path step can
> never be ambiguous? It must reference either an array, optional or scala?
I've confirmed that optionals do not require a predicate, in fact optionals are
not even allowed to have predicates. DAFFODIL-2013 was the issue for this, but
that bug is just wrong--I've added a test to confirm that is the case.
So, I think it is fine for expressions to ambiguously reference a scalar or
optional because they are the same from the infosets perspecive--optionals
might just be missing. But an an expression cannot ambiguously reference an
array or a scala/optional, because the two implementations are completely
different.
One potential issue is what if we have an expression like this:
{code}
fn:count(/ambigous/path/to/scalarOrOptional)
{code}
The fn:count function is supposed to be an SDE if it references a scalar
(DAFFODIL-2711), but maybe if it's ambiguous and it *could* reference an
optional also then we allow it? So fn:count must unambigously reference either
an array or non-array, and if it's a non-array then it must be able to
reference an optional.
> Group evaluation throwing ClassCastException for optional element
> -
>
> Key: DAFFODIL-2879
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2879
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 3.6.0
>Reporter: Peter Katlic
>Priority: Major
>
> A group is defined with an element using fn:count and referenced inside a
> sequence of array and optional elements. See the test count_21b in
> daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml
> for an example.
> Group schema:
> {code:java}
>
>
> dfdl:inputValueCalc="{ fn:count(../ex:foo) }"/>
>
> {code}
> Sequence schema:
> {code:java}
>
>
>
>
>
>
> dfdl:representation="binary"/>
>
>
>
>
>
>
>
> dfdl:representation="binary"/>
>
>
> dfdl:representation="binary" dfdl:inputValueCalc="{ fn:count(../ex:foo) }"/>
>
>
>
>
>
> {code}
> Uncommenting
> {code:java}
> {code}
> and commenting out
> {code:java}
> dfdl:inputValueCalc="{ fn:count(../ex:foo) }"/>{code}
> will produce the following error:
> {code:java}
> java.lang.ClassCastException: class
> org.apache.daffodil.runtime1.infoset.DISimple cannot be cast to class
> org.apache.daffodil.runtime1.infoset.DIArray{code}
> It seems as if the optional is evaluated as a DownArray instead of a
> DownElement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2881) Pin github actions to commit hash instead of tags
[ https://issues.apache.org/jira/browse/DAFFODIL-2881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2881. -- Fix Version/s: 3.7.0 Resolution: Fixed Fixed in commit d5e084c34abbd337d0951a6d33460465e38692e8 > Pin github actions to commit hash instead of tags > - > > Key: DAFFODIL-2881 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2881 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Minor > Fix For: 3.7.0 > > > Github actions should be pinned to a commit hash instead of a version to > prevent malicious actors or just accidental breaking of builds if tags are > renamed/deleted: > https://blog.rafaelgss.dev/why-you-should-pin-actions-by-commit-hash -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2881) Pin github actions to commit hash instead of tags
Steve Lawrence created DAFFODIL-2881: Summary: Pin github actions to commit hash instead of tags Key: DAFFODIL-2881 URL: https://issues.apache.org/jira/browse/DAFFODIL-2881 Project: Daffodil Issue Type: Bug Components: Infrastructure Reporter: Steve Lawrence Assignee: Steve Lawrence Github actions should be pinned to a commit hash instead of a version to prevent malicious actors or just accidental breaking of builds if tags are renamed/deleted: https://blog.rafaelgss.dev/why-you-should-pin-actions-by-commit-hash -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2877) Allow pattern restriction facet to be used on non-string types
[
https://issues.apache.org/jira/browse/DAFFODIL-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821801#comment-17821801
]
Steve Lawrence commented on DAFFODIL-2877:
--
Note that we do already have this problem with xs:strings where Daffodil
converts XML Illegal characters to private use characters. Using a pattern
restriction on those requires that users know to match PUA chars instead of the
chars that are in the data. So this problem isn't really new. It just takes
some education to make users aware that restrictions apply to the canonicalized
infoset representation and not where the data came from.
I agree that schematron or XML Schema 1.1 is probably the best way to do more
complex number checking beyond min/max ranges, but some systems just don't have
access to that.
If we don't want to support this in Daffodil, maybe one option is to change the
SDE to a SDW saying that the pattern restriction is ignored? So we won't
validate patterns with limited validation enabled, but using full validation or
an external validator can still work using the same DFDL schema.
> Allow pattern restriction facet to be used on non-string types
> --
>
> Key: DAFFODIL-2877
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2877
> Project: Daffodil
> Issue Type: New Feature
> Components: Back End, Front End
>Reporter: Steve Lawrence
>Priority: Major
>
> Section 5.2 of the DFDL specification says that the pattern restriction facet
> can only be used for xs:string types. Daffodil currently implements this
> limitation.
> However, there are a number of cases where a pattern is the only way to
> correctly restrict the value of non-string types, such as xs:long.
> One example is a number that can only be between 100 and 300 but in multiples
> of 5. In that case, a pattern like this would work:
> {code}
>
> {code}
> Another example could be apartment rooms, where the first digit represents
> the floor (e.g. 1-5), and the second two digits represent the room number
> between 1 and 15 (assuming 15 rooms per floor). In that case, the pattern
> restriction might be:
> {code}
>
> {code}
> The patterns are often going to be a bit ugly, but in some cases it's the
> only way to correctly validate numbers using only XML Schema capabilities.
> Note that only supporting pattern on xs:string types is a limitation imposed
> by DFDL--XML schema allows pattern restriction on all types. We should remove
> this limitation and allow pattern facet to be used on all types.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2879) Group evaluation throwing ClassCastException for optional element
[
https://issues.apache.org/jira/browse/DAFFODIL-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821414#comment-17821414
]
Steve Lawrence commented on DAFFODIL-2879:
--
I think the issue here is that at one point optionals were implemented as
arrays that could only have a max length of one. I'm not sure when that
changed, but I don't think that is the case anymore. As far as the infoset is
concerned, optionals are the same as scalars that just might not exist.
So I think this ambiguity check might be incorrect:
https://github.com/apache/daffodil/blob/main/daffodil-core/src/main/scala/org/apache/daffodil/core/dpath/Expression.scala#L981-L987
A path step can never ambiguously reference an array or optional, because in
the infoset they are not the same--one is a DIArray and the other is a
DIElement. If anything, we can only allow an ambiguous reference to a scalar or
an optional, since they are both DIElements. And if a step could be an optional
or a scalar, there's just a possibility that the optional might not exist and
create a processing error.
Though, I though we had an issue where optionals required a predicate? So maybe
scalars and optionals can't be allowed either? So maybe a path step can never
be ambiguous? It must reference either an array, optional or scala?
> Group evaluation throwing ClassCastException for optional element
> -
>
> Key: DAFFODIL-2879
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2879
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 3.6.0
>Reporter: Peter Katlic
>Priority: Major
>
> A group is defined with an element using fn:count and referenced inside a
> sequence of array and optional elements. See the test count_21b in
> daffodil-test/src/test/resources/org/apache/daffodil/section23/dfdl_functions/Functions.tdml
> for an example.
> Group schema:
> {code:java}
>
>
> dfdl:inputValueCalc="{ fn:count(../ex:foo) }"/>
>
> {code}
> Sequence schema:
> {code:java}
>
>
>
>
>
>
> dfdl:representation="binary"/>
>
>
>
>
>
>
>
> dfdl:representation="binary"/>
>
>
> dfdl:representation="binary" dfdl:inputValueCalc="{ fn:count(../ex:foo) }"/>
>
>
>
>
>
> {code}
> Uncommenting
> {code:java}
> {code}
> and commenting out
> {code:java}
> dfdl:inputValueCalc="{ fn:count(../ex:foo) }"/>{code}
> will produce the following error:
> {code:java}
> java.lang.ClassCastException: class
> org.apache.daffodil.runtime1.infoset.DISimple cannot be cast to class
> org.apache.daffodil.runtime1.infoset.DIArray{code}
> It seems as if the optional is evaluated as a DownArray instead of a
> DownElement.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DAFFODIL-2880) CLI tunable option does not have an affect using saved parser -- needs error/warning
Steve Lawrence created DAFFODIL-2880: Summary: CLI tunable option does not have an affect using saved parser -- needs error/warning Key: DAFFODIL-2880 URL: https://issues.apache.org/jira/browse/DAFFODIL-2880 Project: Daffodil Issue Type: Bug Components: CLI Reporter: Steve Lawrence When using the CLI with the --parser option to use a saved parser, the -T option to set tunables has no affect. Instead, Daffodil uses the values of tunables set at schema compilation time. This is probably the correct behavior since it could potentially cause odd behavior to have different tunables for schema compilation and schema parsing, but there is no warning or error message that they are ignored and have no affect. So Daffodil just silently ignores the tunable values, which can be confusing if you are expecting a different behavior. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2711) fn:count( non-array ) must issue SDE
[
https://issues.apache.org/jira/browse/DAFFODIL-2711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2711.
--
Fix Version/s: 3.7.0
Resolution: Fixed
Fixed in commit 89d02df846692cda2cde4467e8bc05b6a14251f2
> fn:count( non-array ) must issue SDE
> -
>
> Key: DAFFODIL-2711
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2711
> Project: Daffodil
> Issue Type: Improvement
> Components: Front End
>Affects Versions: 3.3.0
>Reporter: Mike Beckerle
>Assignee: Peter Katlic
>Priority: Major
> Labels: beginner
> Fix For: 3.7.0
>
>
> The fn:count function, when called on a path expression that ends with a
> unique scalar element, should be an SDE.
> Such a call can only return value 1, or cause a processing error if the
> element doesn't exist. Existance should be tested with fn:exists(...).
> If the schema really does have
> {code:java}
>
>
> {code}
> and an expression has `fn:count(/foo)` we already are giving an SDE about
> query-style expressions not being supported.
> If the schema does not have this issue (multiple declarations of same element
> name), then fn:count() makes no sense when called on a scalar, and it should
> be an SDE if the path is not for an array or optional element.
> My experience thusfar is that use of fn:count( ...) with a non-array,
> non-optional element path is always a mistake by the schema author.
> Changing elements from array/optional to scalar is actually fairly common.
> One decides rather than be an array, an element should instead contain the
> array as a child element, but now any fn:count() referring to the
> original array element is incorrect, as it now refers to the scalar element
> that contains the array as a child element. The schema compiler needs to help
> find these erroneous references.
>
> Personally, I think calling fn:count(...) on a non-array/non-optional should
> be an SDE, but
> (This added... Was a fragment sentence initially) ... it may have to be a SDW
> with a tunable to make it an SDE, since we may find out that there are
> schemas depending on this returning 1 when called on scalar elements that
> would be broken by making it an SDE.
> I suggest just making this an SDE first, and then we can see if it needs to
> be backed off to an SDW for some schemas.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2878) cli test expect timeouts need to display the captured text that doesn't meet the expectation
[
https://issues.apache.org/jira/browse/DAFFODIL-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2878.
--
Fix Version/s: 3.7.0
Resolution: Fixed
Fixed in commit 63c4318e9845246cc34a912726d8d0362236f922
> cli test expect timeouts need to display the captured text that doesn't meet
> the expectation
>
>
> Key: DAFFODIL-2878
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2878
> Project: Daffodil
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: 3.6.0
>Reporter: Mike Beckerle
>Priority: Minor
> Fix For: 3.7.0
>
>
> When a CLI test fails due to an expect timeout such as:
>
> {code:java}
> [error] Test
> org.apache.daffodil.cli.cliTest.TestCLIParsing.test_Layer_Execution_Error
> failed: net.sf.expectit.ExpectIOException: Expect operation fails (timeout:
> -1 ms) for matcher: contains('Unexpected exception in layer transformer
> 'buggy''), took 3.414 sec
> [error] at net.sf.expectit.ExpectImpl.expectIn(ExpectImpl.java:106)
> {code}
> It could be good if the actual output text was displayed so that one can see
> why the expectation was not met.
> Without that, it's actually a bit tedious to reproduce the test in order to
> see the actual output. You have to stage daffodil, mine the command line with
> all args from the CLI test, and then issue the command line to daffodil.
> It's not so easy to debug into the expect utility to find the text at issue.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2878) cli test expect timeouts need to display the captured text that doesn't meet the expectation
[
https://issues.apache.org/jira/browse/DAFFODIL-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820201#comment-17820201
]
Steve Lawrence commented on DAFFODIL-2878:
--
I can close the PR if you think "debug = true" is sufficient. It hard to
document it and make it obvious though.
SBT run's tests from the root of the repo, regardless of the subproject (e.g.
daffodil.git).
IDE's run tests from the root of the subproject (e.g.
daffodil.git/daffodil-cli).
I think there are two ways to resolve this:
1. Set the DAFFODIL_HOME environment variable to the absolute path to your
daffodil.git repo. CLI tests will prepend this to all "path()"s, so all your
paths are absolute and the CWD doesn't matter.
2. Modify the IDE so that tests in the daffodil-cli subproject are run from the
daffodil.git directory. I believe [~olabusayo] does this, though I don't know
the details of where the setting is to do this.
Personally I prefer 2, I would like to remove DAFFODIL_HOME so that regardless
of IDE all commands are exactly the same instead of sometimes using relative
paths an sometimes using absolute paths.
> cli test expect timeouts need to display the captured text that doesn't meet
> the expectation
>
>
> Key: DAFFODIL-2878
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2878
> Project: Daffodil
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: 3.6.0
>Reporter: Mike Beckerle
>Priority: Minor
>
> When a CLI test fails due to an expect timeout such as:
>
> {code:java}
> [error] Test
> org.apache.daffodil.cli.cliTest.TestCLIParsing.test_Layer_Execution_Error
> failed: net.sf.expectit.ExpectIOException: Expect operation fails (timeout:
> -1 ms) for matcher: contains('Unexpected exception in layer transformer
> 'buggy''), took 3.414 sec
> [error] at net.sf.expectit.ExpectImpl.expectIn(ExpectImpl.java:106)
> {code}
> It could be good if the actual output text was displayed so that one can see
> why the expectation was not met.
> Without that, it's actually a bit tedious to reproduce the test in order to
> see the actual output. You have to stage daffodil, mine the command line with
> all args from the CLI test, and then issue the command line to daffodil.
> It's not so easy to debug into the expect utility to find the text at issue.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2878) cli test expect timeouts need to display the captured text that doesn't meet the expectation
[
https://issues.apache.org/jira/browse/DAFFODIL-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17820178#comment-17820178
]
Steve Lawrence commented on DAFFODIL-2878:
--
ExpectIOException has a "getInputBuffer" function which I think returns a
string represents what expect is trying to match. Should be a small change.
In the mean time, you should be able to add {{debug = true}} as a parameter to
the to the runCLI function for your test. That will output the CLI command and
echo CLI input and output to stdout.
> cli test expect timeouts need to display the captured text that doesn't meet
> the expectation
>
>
> Key: DAFFODIL-2878
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2878
> Project: Daffodil
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: 3.6.0
>Reporter: Mike Beckerle
>Priority: Minor
>
> When a CLI test fails due to an expect timeout such as:
>
> {code:java}
> [error] Test
> org.apache.daffodil.cli.cliTest.TestCLIParsing.test_Layer_Execution_Error
> failed: net.sf.expectit.ExpectIOException: Expect operation fails (timeout:
> -1 ms) for matcher: contains('Unexpected exception in layer transformer
> 'buggy''), took 3.414 sec
> [error] at net.sf.expectit.ExpectImpl.expectIn(ExpectImpl.java:106)
> {code}
> It could be good if the actual output text was displayed so that one can see
> why the expectation was not met.
> Without that, it's actually a bit tedious to reproduce the test in order to
> see the actual output. You have to stage daffodil, mine the command line with
> all args from the CLI test, and then issue the command line to daffodil.
> It's not so easy to debug into the expect utility to find the text at issue.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DAFFODIL-2877) Allow pattern restriction facet to be used on non-string types
Steve Lawrence created DAFFODIL-2877:
Summary: Allow pattern restriction facet to be used on non-string
types
Key: DAFFODIL-2877
URL: https://issues.apache.org/jira/browse/DAFFODIL-2877
Project: Daffodil
Issue Type: New Feature
Components: Back End, Front End
Reporter: Steve Lawrence
Section 5.2 of the DFDL specification says that the pattern restriction facet
can only be used for xs:string types. Daffodil currently implements this
limitation.
However, there are a number of cases where a pattern is the only way to
correctly restrict the value of non-string types, such as xs:long.
One example is a number that can only be between 100 and 300 but in multiples
of 5. In that case, a pattern like this would work:
{code}
{code}
Another example could be apartment rooms, where the first digit represents the
floor (e.g. 1-5), and the second two digits represent the room number between 1
and 15 (assuming 15 rooms per floor). In that case, the pattern restriction
might be:
{code}
{code}
The patterns are often going to be a bit ugly, but in some cases it's the only
way to correctly validate numbers using only XML Schema capabilities.
Note that only supporting pattern on xs:string types is a limitation imposed by
DFDL--XML schema allows pattern restriction on all types. We should remove this
limitation and allow pattern facet to be used on all types.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2872) Performance for fixed-length textual data formats
[ https://issues.apache.org/jira/browse/DAFFODIL-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17818930#comment-17818930 ] Steve Lawrence commented on DAFFODIL-2872: -- Commit 37fa52e2efed11d4fa272a2f61c47e42302cc6ed improved performance for XML text infoset outputter, it doesn't specifically help this issue, but should help performance of projecting the infoset to XML for all formats > Performance for fixed-length textual data formats > - > > Key: DAFFODIL-2872 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2872 > Project: Daffodil > Issue Type: Improvement > Components: Back End >Affects Versions: 3.6.0 >Reporter: Mike Beckerle >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.7.0 > > > Users have reported very slow performance on schemas for fixed length textual > formats such as ARINC 484. (132 column lines, fields are fixed length) > It is reported that it takes minutes to parse 5 million lines. Despite > changing the schema to use choice-by-dispatch everywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2875) Incorrect warning about ignored properties
Steve Lawrence created DAFFODIL-2875:
Summary: Incorrect warning about ignored properties
Key: DAFFODIL-2875
URL: https://issues.apache.org/jira/browse/DAFFODIL-2875
Project: Daffodil
Issue Type: Bug
Components: Front End
Reporter: Steve Lawrence
Say we have this schema:
{code:xml}
http://www.ogf.org/dfdl/dfdl-1.0/;
xmlns:xs="http://www.w3.org/2001/XMLSchema;>
http://www.ogf.org/dfdl/;>
{code}
When compiling this schema we get the warnings:
{quote}
[warn] Schema Definition Warning: DFDL property was ignored:
textStandardBase="10" (id: ignoreDFDLProperty)
Schema context: Day Location line 52 column 4 in
.../home/slawrence/downloads/foo.dfdl.xsd
[warn] Schema Definition Warning: DFDL property was ignored:
textNumberPattern="00" (id: ignoreDFDLProperty)
Schema context: Day Location line 52 column 4 in
.../home/slawrence/downloads/foo.dfdl.xsd
[warn] Schema Definition Warning: DFDL property was ignored:
textNumberRounding="pattern" (id: ignoreDFDLProperty)
Schema context: Day Location line 52 column 4 in
.../home/slawrence/downloads/foo.dfdl.xsd
[warn] Schema Definition Warning: DFDL property was ignored:
textNumberRep="standard" (id: ignoreDFDLProperty)
Schema context: Day Location line 52 column 4 in
.../home/slawrence/downloads/foo.dfdl.xsd
[warn] Schema Definition Warning: DFDL property was ignored:
textNumberCheckPolicy="strict" (id: ignoreDFDLProperty)
Schema context: Day Location line 52 column 4 in
.../home/slawrence/downloads/foo.dfdl.xsd
{quote}
Testing shows this warnings are clearly wrong and are not actually ignored.
One workaround is to use a separate simpleType for your elements and use that
as the type of your referenced element. For example, change your Day element to
this:
{code:xml}
{code}
For reference, here's what I think is going on:
The schema has multiple references to the same elements (e.g. two references to
the Day element). This allows Daffodil to share internal parsers. So even
though there are two references to the "Day" element, we only generate one
"Day" sub-parser, and reuse the exact same sub-parser in two different places
in the final parser we generate. This can pretty drastically reduce schema
compilation time because we can completely skip building a second sub-parser
that is identical.
The problem is that we store which properties an element uses on the local
element that does the referencing, not the element that is actually referenced.
So we have two different lists of "used properties".
The first local element generates a sub-parser for the referenced element, and
that uses all of the properties, and marks the properties as used by the first
local element.
The second local element figures out that it can share the sub-parser, so any
properties that are used when that sub-parser is generated do not get marked as
used on the second local element.
And then when we do our check for unused properties, from the perspective of
the second local element, some properties were never used because it doesn't
know the first local element used them and the second element skipped all that
work.
And I think the workaround works because the referenced element has no
properties defined directly on it so it looks like there are no properties that
might be unused. This might actually be a related bug, in that we should be
considering if the properties defined on the simple type of the referenced
element are used or not. With the mentioned workaround, I've confirmed that if
the simple type has properties that are not used then there is no warning like
there should be.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2812) Exclude xml-apis from transitive dependencies
[
https://issues.apache.org/jira/browse/DAFFODIL-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17808196#comment-17808196
]
Steve Lawrence commented on DAFFODIL-2812:
--
I tried to excluded xml-apis as a dependency from xerces by changing the
depending line to this:
{code:scala}
"xerces" % "xercesImpl" % "2.12.2" exclude("xml-apis", "xml-apis"),
{code}
It compiled fine but running tests on Java 8 fail with this stack trace:
{code:java}
[error] java.lang.NoClassDefFoundError: org/w3c/dom/ElementTraversal
[error] at java.lang.ClassLoader.defineClass1(Native Method)
[error] at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
[error] at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
[error] at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
[error] at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
[error] at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
[error] at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
[error] at java.security.AccessController.doPrivileged(Native Method)
[error] at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
[error] at
sbt.internal.ManagedClassLoader.findClass(ManagedClassLoader.java:103)
[error] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
[error] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
[error] at java.lang.ClassLoader.defineClass1(Native Method)
[error] at java.lang.ClassLoader.defineClass(ClassLoader.java:756)
[error] at
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
[error] at java.net.URLClassLoader.defineClass(URLClassLoader.java:473)
[error] at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
[error] at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
[error] at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
[error] at java.security.AccessController.doPrivileged(Native Method)
[error] at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
[error] at
sbt.internal.ManagedClassLoader.findClass(ManagedClassLoader.java:103)
[error] at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
[error] at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
[error] at
org.apache.xerces.jaxp.DocumentBuilderImpl.newDocument(Unknown Source)
[error] at
org.apache.daffodil.runtime1.infoset.W3CDOMInfosetOutputter.startDocument(W3CDOMInfosetOutputter.scala:52)
[error] at
org.apache.daffodil.runtime1.infoset.TeeInfosetOutputter.$anonfun$startDocument$1(TeeInfosetOutputter.scala:64)
{code}
All tests pass when running on Java 11, so maybe xml-apis wasn't fully moved in
the Java std library until at some point after Java 8? There have been
discussions about dropping Java 8 support, so maybe once that is done we can
remove the xml-apis dependency?
> Exclude xml-apis from transitive dependencies
> -
>
> Key: DAFFODIL-2812
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2812
> Project: Daffodil
> Issue Type: Improvement
> Components: Libraries
>Affects Versions: 3.4.0
>Reporter: John Interrante
>Assignee: Steve Lawrence
>Priority: Minor
> Fix For: 3.7.0
>
>
> Xerces doesn't actually need xml-apis for the same reason that Saxon doesn't
> need it. The xml-apis library is used to provide a consistent set of XML
> APIs across different Java versions and implementations, but the XML APIs are
> included in the standard Java runtime library in all current Java LTS
> releases (8, 11, 17). We very likely can exclude xml-apis from Daffodil's
> dependencies safely, although we should do a lot of testing in diverse
> environments because the whole point of xml-apis is to provide a consistent
> set of XML APIs in these diverse environments. Still, Saxon also works in
> many diverse environments so if it can exclude xml-apis safely in its most
> recent 12.2 release, it seems that Daffodil can exclude xml-apis safely too.
> Note that Java 8 and Java 11/17 differ in how they package the Java runtime
> library. Java 11/17 have added module paths as well as class paths and moved
> the XML APIs into the standard Java module [java.xml (Java SE 11 & JDK 11 )
> (oracle.com)|https://docs.oracle.com/en/java/javase/11/docs/api/java.xml/module-summary.html].
> I do not believe Java's standard module path will cause a problem for
> Daffodil or Java applications embedding Daffodil libraries, but Java
> applications which use a smaller custom module path may need to add the
> java.xml module.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2849) Daffodil jars in RPM/tar/zip do not have the same name as those published to maven
[ https://issues.apache.org/jira/browse/DAFFODIL-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2849. -- Resolution: Fixed Fixed in commit fd7005995850cb72af9a58e4eb1d38b30db183a8 > Daffodil jars in RPM/tar/zip do not have the same name as those published to > maven > -- > > Key: DAFFODIL-2849 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2849 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Priority: Major > Fix For: 3.7.0 > > > When sbt native packager is used to create an RPM, tar, or zip release files, > the daffodil jars do not have the "_2.12" in the name. This usually isn't a > big deal because we package these files along with the scala library, so we > know they are compatible. > However, this can make it a bit more difficult to verify the published maven > artifacts are the same as those in the helper binaries since they have > different names. I have also heard of instances where the files in the RPM > are used to create a mini local maven repository, which requires changing pom > dependencies because the _2.12 is missing. > The sbt native packager plugin is used to creae the RPM, tar, and zip > files--we should see if it can be configured so that it will include the > _2.12 version in the jar names. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DAFFODIL-2849) Daffodil jars in RPM/tar/zip do not have the same name as those published to maven
[ https://issues.apache.org/jira/browse/DAFFODIL-2849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence updated DAFFODIL-2849: - Fix Version/s: 3.7.0 > Daffodil jars in RPM/tar/zip do not have the same name as those published to > maven > -- > > Key: DAFFODIL-2849 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2849 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Priority: Major > Fix For: 3.7.0 > > > When sbt native packager is used to create an RPM, tar, or zip release files, > the daffodil jars do not have the "_2.12" in the name. This usually isn't a > big deal because we package these files along with the scala library, so we > know they are compatible. > However, this can make it a bit more difficult to verify the published maven > artifacts are the same as those in the helper binaries since they have > different names. I have also heard of instances where the files in the RPM > are used to create a mini local maven repository, which requires changing pom > dependencies because the _2.12 is missing. > The sbt native packager plugin is used to creae the RPM, tar, and zip > files--we should see if it can be configured so that it will include the > _2.12 version in the jar names. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2871) No way to pad after affix if positive pattern does not have an affix in textNumberPattern
[ https://issues.apache.org/jira/browse/DAFFODIL-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2871. -- Resolution: Fixed Fixed in commit cdb41212d755c45f23f981c395dd24ab56303b45 > No way to pad after affix if positive pattern does not have an affix in > textNumberPattern > - > > Key: DAFFODIL-2871 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2871 > Project: Daffodil > Issue Type: Bug > Components: Front End >Affects Versions: 3.6.0 >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.7.0 > > > ICU only uses the positive pattern of the textNumberPattern property to > determine pad character and position, completely ignoring the negative > pattern. If the positive pattern has no affix associated with the pad > character, then there is an ambiguity if the pad character should appear > before or after the negative affix when unparsing negative numbers. In this > case, ICU defaults to before the affix, with no way to change it via the > pattern. This effectively means it is not possible for ICU number padding to > appear after a negative affix if there is no positive affix. > We need a way to specify that ICU pad characters should apper after an affix > if the positive pattern does not have an affix. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2856) Configure SonarCloud scan run on Java 17
[
https://issues.apache.org/jira/browse/DAFFODIL-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2856.
--
Resolution: Fixed
Fixed in commit e2a1835b7d23057f4916ae50b8d08b605ecf714e
> Configure SonarCloud scan run on Java 17
>
>
> Key: DAFFODIL-2856
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2856
> Project: Daffodil
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Steve Lawrence
>Assignee: Peter Katlic
>Priority: Critical
> Fix For: 3.7.0
>
>
> Received an email with the following from SonarCloud:
> {quote}
> Starting from the 15th of November 2023, SonarCloud will no longer accept
> scans executed using Java 11. This decision has been made to ensure the
> platform's compatibility with the latest technologies and to provide you with
> an enhanced experience.
> To continue benefiting from SonarCloud's advanced features and analysis
> capabilities, we strongly encourage you to migrate your runtime environment
> for the analysis to Java 17 as soon as possible.{quote}
> We should update our github action to run on our Java 17 matrix instead of
> Java 11:
> https://github.com/apache/daffodil/blob/main/.github/workflows/main.yml#L88-L95
> Marking as critical since this will happen fairly soon, and we don't want
> builds to start breaking.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2856) Configure SonarCloud scan run on Java 17
[
https://issues.apache.org/jira/browse/DAFFODIL-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17807356#comment-17807356
]
Steve Lawrence commented on DAFFODIL-2856:
--
According to this page:
https://community.sonarsource.com/t/java-11-is-deprecated-as-a-runtime-env-to-scan-your-projects/96597
We need to use version 2.0.0 or newer of the sonarcloud-github-action. We are
currently on 1.9, so that might explain the problem. Dependabot should have
been keeping these versions up to date, but maybe that's broken? It would be
nice to get that fixed, but in the meantime we should manually update this one.
> Configure SonarCloud scan run on Java 17
>
>
> Key: DAFFODIL-2856
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2856
> Project: Daffodil
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Steve Lawrence
>Assignee: Peter Katlic
>Priority: Critical
> Fix For: 3.7.0
>
>
> Received an email with the following from SonarCloud:
> {quote}
> Starting from the 15th of November 2023, SonarCloud will no longer accept
> scans executed using Java 11. This decision has been made to ensure the
> platform's compatibility with the latest technologies and to provide you with
> an enhanced experience.
> To continue benefiting from SonarCloud's advanced features and analysis
> capabilities, we strongly encourage you to migrate your runtime environment
> for the analysis to Java 17 as soon as possible.{quote}
> We should update our github action to run on our Java 17 matrix instead of
> Java 11:
> https://github.com/apache/daffodil/blob/main/.github/workflows/main.yml#L88-L95
> Marking as critical since this will happen fairly soon, and we don't want
> builds to start breaking.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (DAFFODIL-2856) Configure SonarCloud scan run on Java 17
[
https://issues.apache.org/jira/browse/DAFFODIL-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence reopened DAFFODIL-2856:
--
Merged PR's are starting to fail on the SonarCloud Scan job with the error:
{quote}
The version of Java (11.0.20) used to run this analysis is deprecated, and
SonarCloud no longer supports it. Please upgrade to Java 17 or later.
As a temporary measure, you can set the property
'sonar.scanner.force-deprecated-java-version' to 'true' to continue using Java
11.0.20
This workaround will only be effective until January 28, 2024. After this date,
all scans using the deprecated Java 11 will fail.
{quote}
The job is running with Java 17, so it's not clear where it is getting java 11
from.
> Configure SonarCloud scan run on Java 17
>
>
> Key: DAFFODIL-2856
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2856
> Project: Daffodil
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Steve Lawrence
>Assignee: Peter Katlic
>Priority: Critical
> Fix For: 3.7.0
>
>
> Received an email with the following from SonarCloud:
> {quote}
> Starting from the 15th of November 2023, SonarCloud will no longer accept
> scans executed using Java 11. This decision has been made to ensure the
> platform's compatibility with the latest technologies and to provide you with
> an enhanced experience.
> To continue benefiting from SonarCloud's advanced features and analysis
> capabilities, we strongly encourage you to migrate your runtime environment
> for the analysis to Java 17 as soon as possible.{quote}
> We should update our github action to run on our Java 17 matrix instead of
> Java 11:
> https://github.com/apache/daffodil/blob/main/.github/workflows/main.yml#L88-L95
> Marking as critical since this will happen fairly soon, and we don't want
> builds to start breaking.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DAFFODIL-2871) No way to pad after affix if positive pattern does not have an affix in textNumberPattern
Steve Lawrence created DAFFODIL-2871: Summary: No way to pad after affix if positive pattern does not have an affix in textNumberPattern Key: DAFFODIL-2871 URL: https://issues.apache.org/jira/browse/DAFFODIL-2871 Project: Daffodil Issue Type: Bug Components: Front End Affects Versions: 3.6.0 Reporter: Steve Lawrence Assignee: Steve Lawrence Fix For: 3.7.0 ICU only uses the positive pattern of the textNumberPattern property to determine pad character and position, completely ignoring the negative pattern. If the positive pattern has no affix associated with the pad character, then there is an ambiguity if the pad character should appear before or after the negative affix when unparsing negative numbers. In this case, ICU defaults to before the affix, with no way to change it via the pattern. This effectively means it is not possible for ICU number padding to appear after a negative affix if there is no positive affix. We need a way to specify that ICU pad characters should apper after an affix if the positive pattern does not have an affix. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DAFFODIL-2869) Update copyright year in NOTICE and website
[ https://issues.apache.org/jira/browse/DAFFODIL-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence updated DAFFODIL-2869: - Summary: Update copyright year in NOTICE and website (was: Update copyright year in NOTICE and websie) > Update copyright year in NOTICE and website > --- > > Key: DAFFODIL-2869 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2869 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.7.0 > > > The copyright year in our NOTICE files and website hasn't been updated since > 2022. It's now 2024, we should update it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2869) Update copyright year in NOTICE and websie
[ https://issues.apache.org/jira/browse/DAFFODIL-2869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2869. -- Fix Version/s: 3.7.0 Resolution: Fixed Daffodil fixed in commit 9e0bdf67d4b264732e52ce6f21a0ba675fc3bace Website fixed in commit 50801b0f08a8b14bb6f3bddf1d2d717c74fadb4b > Update copyright year in NOTICE and websie > -- > > Key: DAFFODIL-2869 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2869 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.7.0 > > > The copyright year in our NOTICE files and website hasn't been updated since > 2022. It's now 2024, we should update it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2868) Slow removal of string padding
[ https://issues.apache.org/jira/browse/DAFFODIL-2868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2868. -- Resolution: Fixed Fixed in commit f4996f0fe86643e9282d37a06263c3cc5a82de81 > Slow removal of string padding > -- > > Key: DAFFODIL-2868 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2868 > Project: Daffodil > Issue Type: Bug > Components: Back End, Performance >Reporter: Steve Lawrence >Priority: Major > Fix For: 3.7.0 > > > The code to remove padding is clear and concise but it fairly inefficient, > enough to show up in profiling. We should optimize this algorithm to be more > efficient and reduce allocations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2869) Update copyright year in NOTICE and websie
Steve Lawrence created DAFFODIL-2869: Summary: Update copyright year in NOTICE and websie Key: DAFFODIL-2869 URL: https://issues.apache.org/jira/browse/DAFFODIL-2869 Project: Daffodil Issue Type: Bug Components: Infrastructure Reporter: Steve Lawrence Assignee: Steve Lawrence The copyright year in our NOTICE files and website hasn't been updated since 2022. It's now 2024, we should update it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2868) Slow removal of string padding
Steve Lawrence created DAFFODIL-2868: Summary: Slow removal of string padding Key: DAFFODIL-2868 URL: https://issues.apache.org/jira/browse/DAFFODIL-2868 Project: Daffodil Issue Type: Bug Components: Back End, Performance Reporter: Steve Lawrence Fix For: 3.7.0 The code to remove padding is clear and concise but it fairly inefficient, enough to show up in profiling. We should optimize this algorithm to be more efficient and reduce allocations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2867) Parsing of xs:float as text failure
[ https://issues.apache.org/jira/browse/DAFFODIL-2867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2867. -- Fix Version/s: 3.7.0 Resolution: Fixed Fixed in commit 20d6ea3479c85e431faa260424ecb5e2675d7d8c > Parsing of xs:float as text failure > --- > > Key: DAFFODIL-2867 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2867 > Project: Daffodil > Issue Type: Bug > Components: Back End >Affects Versions: 3.6.0 >Reporter: Mike Beckerle >Priority: Major > Fix For: 3.7.0 > > > > Bug reproduction details forthcoming shortly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-790) Failure to reset global debugger settings causes random tests to fail in SBT.
[ https://issues.apache.org/jira/browse/DAFFODIL-790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-790. - Resolution: Fixed Verified this is fixed and updated a test to confirm in commit f41f06ae8b6eec30b3b41fff026f623c7a80cd30 > Failure to reset global debugger settings causes random tests to fail in SBT. > - > > Key: DAFFODIL-790 > URL: https://issues.apache.org/jira/browse/DAFFODIL-790 > Project: Daffodil > Issue Type: Bug > Components: General, Infrastructure >Reporter: Taylor Wise >Assignee: Mike Beckerle >Priority: Minor > Labels: dev-verify, thread-safety > Fix For: 3.7.0 > > > To demonstrate this create a test. Before the runOneTest command create and > add a Debugger. Do not reset the Debugger to the global defaults at the end > of the test. > As an example look in TestUserSubmittedTests.scala at test_dfdl_782. Comment > out the last two Debugger lines after runner.runOneTest. This will cause all > sorts of failures in SBT. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2861) Assertion thrown when encoding error with dfdl:encodingErrorPolicy="error"
[
https://issues.apache.org/jira/browse/DAFFODIL-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2861.
--
Fix Version/s: 3.7.0
Resolution: Fixed
Fixed in commit 37d6746a7d2d52dabd0697dcc178067fad814cea
> Assertion thrown when encoding error with dfdl:encodingErrorPolicy="error"
> --
>
> Key: DAFFODIL-2861
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2861
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 3.0.0
>Reporter: Steve Lawrence
>Priority: Major
> Fix For: 3.7.0
>
>
> If a schema sets {{dfdl:encodingErrorPolicy="error"}}, then Daffodil outputs
> this warning:
> {quote}
> [warning] Schema Definition Warning: dfdl:encodingErrorPolicy="error" is not
> yet implemented. The 'replace' value will be used.
> {quote}
> The idea is we want to be portable with schemas written for IBM DFDL (which
> only supports "error" and not "replace"), so we support the "error" property
> value but use the our only supported "replace" behavior. In the common case
> where there are no encoding errors, the behavior will be exactly the same as
> IBM DFDL.
> But if there is an encoding error, then Daffodil throws this assertion:
> {code}
> !!
> !! Not Yet Implemented !!
> !!
> You are using a feature that is not yet implemented:
> Some(Not yet implemented: dfdl:encodingErrorPolicy="error"
> org.apache.daffodil.lib.exceptions.Assert$.nyi(Assert.scala:151)
> org.apache.daffodil.io.processors.charset.BitsCharsetDecoder.decodeOneHandleMalformed(BitsCharsetDecoder.scala:122)
> org.apache.daffodil.io.processors.charset.BitsCharsetDecoder.decode(BitsCharsetDecoder.scala:84)
> org.apache.daffodil.io.InputSourceDataInputStreamCharIterator.fetch(InputSourceDataInputStream.scala:906))
> {code}
> So even though the warning says we are going to use "replace", we still
> clearly use "error", which results in an assertion failure that we don't
> support it.
> I confirmed this occurs at least as far back as Daffodil 3.0.0.
> Here is a schema that reproduces the issue:
> {code:xml}
>schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormatPortable.dfdl.xsd"
> />
>
> http://www.ogf.org/dfdl/;>
>
>
>
>dfdl:lengthKind="delimited" dfdl:terminator="%NL;"
> dfdl:encoding="ASCII" />
> {code}
> Any file that non-ASCII characters followed by a new line should trigger this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DAFFODIL-2753) Remove numbers from CLI integration test names
[ https://issues.apache.org/jira/browse/DAFFODIL-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence updated DAFFODIL-2753: - Fix Version/s: 3.7.0 > Remove numbers from CLI integration test names > -- > > Key: DAFFODIL-2753 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2753 > Project: Daffodil > Issue Type: Improvement > Components: Clean Ups >Reporter: Steve Lawrence >Assignee: Peter Katlic >Priority: Trivial > Labels: beginner > Fix For: 3.7.0 > > > Many of the integration tests include a seemingly random number (e.g. > test_3677_CLI_Parsing_elementFormDefault_qualified). These numbers actually > come from an old requirements document that no longer exists or is > maintained, and so just causes confusion about what they might mean. We > should just remove these numbers to avoid potential confusion. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2823) time zone "z" specifier does not work properly
[ https://issues.apache.org/jira/browse/DAFFODIL-2823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2823. -- Resolution: Not A Problem > time zone "z" specifier does not work properly > -- > > Key: DAFFODIL-2823 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2823 > Project: Daffodil > Issue Type: Bug > Components: Back End >Affects Versions: 3.4.0 >Reporter: Mike Beckerle >Priority: Minor > Fix For: 3.5.0 > > > The time zone string "EST" is parsed by calendarPattern charater "z", and > becomes ISO standard "-05:00" on parse, but "GMT-5" on unparse. > This happens despite dfdl:calendarLanguge="en_US". That is, the problem is > that the unparse should produce "EST", but does not. > The locale is needed to implement "z" time zone format because some of these > 3-letter timezone specifiers are ambiguous, so the locale must be known to > disambiguate them when parsing. > For example CST can be "Central Standard Time" (North America), "Cuba > Standard Time" or "China Standard Time", and is widely used for "Central > Standard Time" (Australia) according to > [https://en.wikipedia.org/wiki/List_of_time_zone_abbreviations] > So when parsing "z" one must use the locale. When unparsing it's not so > ambiguous, but Daffodil doesn't appear to use the locale information. > Per the ICU documentation here: > [https://icu.unicode.org/design/formatting/timezone/icu-4-8-time-zone-names] > The "z" needs locale information, and several of the other specifiers also > need it. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DAFFODIL-2823) time zone "z" specifier does not work properly
[ https://issues.apache.org/jira/browse/DAFFODIL-2823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence closed DAFFODIL-2823. > time zone "z" specifier does not work properly > -- > > Key: DAFFODIL-2823 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2823 > Project: Daffodil > Issue Type: Bug > Components: Back End >Affects Versions: 3.4.0 >Reporter: Mike Beckerle >Priority: Minor > Fix For: 3.5.0 > > > The time zone string "EST" is parsed by calendarPattern charater "z", and > becomes ISO standard "-05:00" on parse, but "GMT-5" on unparse. > This happens despite dfdl:calendarLanguge="en_US". That is, the problem is > that the unparse should produce "EST", but does not. > The locale is needed to implement "z" time zone format because some of these > 3-letter timezone specifiers are ambiguous, so the locale must be known to > disambiguate them when parsing. > For example CST can be "Central Standard Time" (North America), "Cuba > Standard Time" or "China Standard Time", and is widely used for "Central > Standard Time" (Australia) according to > [https://en.wikipedia.org/wiki/List_of_time_zone_abbreviations] > So when parsing "z" one must use the locale. When unparsing it's not so > ambiguous, but Daffodil doesn't appear to use the locale information. > Per the ICU documentation here: > [https://icu.unicode.org/design/formatting/timezone/icu-4-8-time-zone-names] > The "z" needs locale information, and several of the other specifiers also > need it. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2823) time zone "z" specifier does not work properly
[ https://issues.apache.org/jira/browse/DAFFODIL-2823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17795393#comment-17795393 ] Steve Lawrence commented on DAFFODIL-2823: -- The new API is the addition of the "getIanaID" function: https://unicode-org.github.io/icu-docs/apidoc/dev/icu4j/com/ibm/icu/util/TimeZone.html#getIanaID-java.lang.String- This just uses a newer database for getting a canonical timezone ID, and doesn't help the issue of picking the right timezone from a UTF offset (which mentioned above is hard because there isn't really a "right" timezone. Reclosing this as "Not a Problem" since this new API doesn't provide a solution to this issue, and this is just how things work. > time zone "z" specifier does not work properly > -- > > Key: DAFFODIL-2823 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2823 > Project: Daffodil > Issue Type: Bug > Components: Back End >Affects Versions: 3.4.0 >Reporter: Mike Beckerle >Priority: Minor > Fix For: 3.5.0 > > > The time zone string "EST" is parsed by calendarPattern charater "z", and > becomes ISO standard "-05:00" on parse, but "GMT-5" on unparse. > This happens despite dfdl:calendarLanguge="en_US". That is, the problem is > that the unparse should produce "EST", but does not. > The locale is needed to implement "z" time zone format because some of these > 3-letter timezone specifiers are ambiguous, so the locale must be known to > disambiguate them when parsing. > For example CST can be "Central Standard Time" (North America), "Cuba > Standard Time" or "China Standard Time", and is widely used for "Central > Standard Time" (Australia) according to > [https://en.wikipedia.org/wiki/List_of_time_zone_abbreviations] > So when parsing "z" one must use the locale. When unparsing it's not so > ambiguous, but Daffodil doesn't appear to use the locale information. > Per the ICU documentation here: > [https://icu.unicode.org/design/formatting/timezone/icu-4-8-time-zone-names] > The "z" needs locale information, and several of the other specifiers also > need it. > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-697) Java API: ProcessorFactory's setDistinguishedRootNode method does not appear to work.
[ https://issues.apache.org/jira/browse/DAFFODIL-697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-697. - Fix Version/s: 3.7.0 Assignee: Steve Lawrence Resolution: Fixed Fixed in commit b8976946ec02c0475ce665b527db3d6c7014dba9 > Java API: ProcessorFactory's setDistinguishedRootNode method does not appear > to work. > - > > Key: DAFFODIL-697 > URL: https://issues.apache.org/jira/browse/DAFFODIL-697 > Project: Daffodil > Issue Type: Bug > Components: API >Reporter: Jessie Chab >Assignee: Steve Lawrence >Priority: Minor > Fix For: 3.7.0 > > > I copied testJavaAPI3 in > daffodil-core/src/test/java/edu/illinois/ncsa/daffodil/example/TestJavaAPI.java > and added a new element to > daffodil-core/src/test/resources/test/japi/mySchema3.dfdl.xsd > I modified the test (new test is testJavaAPI4) to reference my new element > (e4) and my new data file (myData2.bat) > When I run the test it still seems to be referencing element e3, the first > element in the schema. When I swap e3 and e4 in the schema, all tests that > reference that schema use e4 - the first element. > Steve suggested trying the compiler's setDistinguishedRootNode before running > compile, and that worked with no problems. It only seems to be the > ProcessorFactory's setDistinguishedRootNode method that isn't working. > I commented out the failing test (testJavaAPI4) and added the passing test > that uses the compiler's method (testJavaAPI4b) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2864) Regression in distinguished root node
[
https://issues.apache.org/jira/browse/DAFFODIL-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2864.
--
Fix Version/s: 3.7.0
Assignee: Steve Lawrence
Resolution: Fixed
Fixed in commit b8976946ec02c0475ce665b527db3d6c7014dba9
> Regression in distinguished root node
> -
>
> Key: DAFFODIL-2864
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2864
> Project: Daffodil
> Issue Type: Bug
>Affects Versions: 3.5.0, 3.6.0
>Reporter: Claude Mamo
>Assignee: Steve Lawrence
>Priority: Major
> Fix For: 3.7.0
>
>
> We've updated from Daffodil 3.4 to 3.5. Following the update, one of our
> Smooks examples started failing. It appears that the distinguished root node
> set from the processor factory is now being ignored. The same behaviour
> occurs on Daffodil 3.6. I've created a test case which reproduces the
> unexpected behaviour here:
> [https://github.com/claudemamo/smooks-dfdl-cartridge/tree/daffodil-distinguishedRootNode-issue].
> Running the following will give you an error:
> {code:java}
> mvn clean test
> -Dtest=org.smooks.cartridges.dfdl.FunctionalTestCase#testSmooksConfigGivenDistinguishedRootNode{code}
>
> The error is:
> {quote}
> Unparse Error: Expected element start event for ex:\{http://example.com}file,
> but received element start event for (invalid) \{http://example.com}record
> {quote}
> The expected start event should actually be
> _[http://example.com|http://example.com/]:record_ because this is what the
> distinguished root node is set to. The distinguished root node is set in this
> line of code:
> [https://github.com/claudemamo/smooks-dfdl-cartridge/blob/daffodil-distinguishedRootNode-issue/src/main/java/org/smooks/cartridges/dfdl/DfdlSchema.java#L154].
> I've checked the release notes for Daffodil 3.5 and nothing stands out. The
> test case passes on Daffodil 3.4. Any advice on what the problem could be?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2864) Regression in distinguished root node
[
https://issues.apache.org/jira/browse/DAFFODIL-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17790574#comment-17790574
]
Steve Lawrence commented on DAFFODIL-2864:
--
This might be related to DAFFODIL-697, which talks about setDistinguishedRoot
node not working correctly. That function has been long deprecated and removed,
but maybe has a similar underlying issue to withDistinguishedRoot.
However, for the 3.5 and 3.6 releases, a workaround should be to provide the
root to the {{compileSource}} function via the optional name/namespace
parameters. Something like this should work:
{code:java}
String rootName = null;
String rootNamespace = null;
if (distinguishedRootNode != null) {
rootName = ... // get from distinguishedRootNode
rootNamespace = ... // get from distingushedRootNode
}
ProcessorFactory processorFactory = compiler.compileSource(uri, rootName,
rootNamespace);
if (processorFactory.isError) {
...
}
{code}
In fact, this workaround is probably the better way to do it. When you call
{{compileSource}} without a name/namespace, I believe Daffodil does some amount
of work using the first element it sees in the schema (this might be a bug,
maybe we shouldn't do any work until the user calls pf.isError?). If you then
call ProcessorFactory.withDistinguishedRootNode, we're going to have to redo
some of that work with the new root node. So the previous work was just wasted
effort. I think the issue right now is that we just don't redo that work with
the new root, which is why this function has no effect. That said, the code
around this in 3.4 looks very similar to 3.5 and 3.6, so maybe this hunch is
wrong. Regardless, the above workaround should work as expected.
I'll continue to look into this to figure out the underlying cause.
> Regression in distinguished root node
> -
>
> Key: DAFFODIL-2864
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2864
> Project: Daffodil
> Issue Type: Bug
>Affects Versions: 3.5.0, 3.6.0
>Reporter: Claude Mamo
>Priority: Major
>
> We've updated from Daffodil 3.4 to 3.5. Following the update, one of our
> Smooks examples started failing. It appears that the distinguished root node
> set from the processor factory is now being ignored. The same behaviour
> occurs on Daffodil 3.6. I've created a test case which reproduces the
> unexpected behaviour here:
> [https://github.com/claudemamo/smooks-dfdl-cartridge/tree/daffodil-distinguishedRootNode-issue.]
> Running the following will give you an error:
> {code:java}
> mvn clean test
> -Dtest=org.smooks.cartridges.dfdl.FunctionalTestCase#testSmooksConfigGivenDistinguishedRootNode{code}
>
> The error is:
> {quote}
> Unparse Error: Expected element start event for ex:\{http://example.com}file,
> but received element start event for (invalid) \{http://example.com}record
> {quote}
> The expected start event should actually be
> _[http://example.com|http://example.com/]:record_ because this is what the
> distinguished root node is set to. The distinguished root node is set in this
> line of code:
> [https://github.com/claudemamo/smooks-dfdl-cartridge/blob/daffodil-distinguishedRootNode-issue/src/main/java/org/smooks/cartridges/dfdl/DfdlSchema.java#L154].
> I've checked the release notes for Daffodil 3.5 and nothing stands out. The
> test case passes on Daffodil 3.4. Any advice on what the problem could be?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (DAFFODIL-2823) time zone "z" specifier does not work properly
[
https://issues.apache.org/jira/browse/DAFFODIL-2823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence reopened DAFFODIL-2823:
--
ICU4J 74.1 mentions this in the release notes:
{quote}New TimeZone API for getting the “primary” IANA time zone ID, rather
than the CLDR-canonical ID. (ICU-22452){quote}
It's possible this new API allows the "z" specifier to work as one might
expected.
I'm going to reopen this so we can investigate this new timezone API and see if
it makes a difference. If not, we can re-close as no-a-problem.
> time zone "z" specifier does not work properly
> --
>
> Key: DAFFODIL-2823
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2823
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 3.4.0
>Reporter: Mike Beckerle
>Priority: Minor
> Fix For: 3.5.0
>
>
> The time zone string "EST" is parsed by calendarPattern charater "z", and
> becomes ISO standard "-05:00" on parse, but "GMT-5" on unparse.
> This happens despite dfdl:calendarLanguge="en_US". That is, the problem is
> that the unparse should produce "EST", but does not.
> The locale is needed to implement "z" time zone format because some of these
> 3-letter timezone specifiers are ambiguous, so the locale must be known to
> disambiguate them when parsing.
> For example CST can be "Central Standard Time" (North America), "Cuba
> Standard Time" or "China Standard Time", and is widely used for "Central
> Standard Time" (Australia) according to
> [https://en.wikipedia.org/wiki/List_of_time_zone_abbreviations]
> So when parsing "z" one must use the locale. When unparsing it's not so
> ambiguous, but Daffodil doesn't appear to use the locale information.
> Per the ICU documentation here:
> [https://icu.unicode.org/design/formatting/timezone/icu-4-8-time-zone-names]
> The "z" needs locale information, and several of the other specifiers also
> need it.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2862) Prepare for 3.7.0 development
[ https://issues.apache.org/jira/browse/DAFFODIL-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2862. -- Resolution: Fixed Fixed in commit f1cde32e5b2439e045d6b5047661299c3f01f752 > Prepare for 3.7.0 development > - > > Key: DAFFODIL-2862 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2862 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (DAFFODIL-2862) Prepare for 3.7.0 development
[ https://issues.apache.org/jira/browse/DAFFODIL-2862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence updated DAFFODIL-2862: - Fix Version/s: 3.7.0 > Prepare for 3.7.0 development > - > > Key: DAFFODIL-2862 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2862 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Priority: Major > Fix For: 3.7.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2862) Prepare for 3.7.0 development
Steve Lawrence created DAFFODIL-2862: Summary: Prepare for 3.7.0 development Key: DAFFODIL-2862 URL: https://issues.apache.org/jira/browse/DAFFODIL-2862 Project: Daffodil Issue Type: Bug Components: Infrastructure Reporter: Steve Lawrence -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2861) Assertion thrown when encoding error with dfdl:encodingErrorPolicy="error"
Steve Lawrence created DAFFODIL-2861:
Summary: Assertion thrown when encoding error with
dfdl:encodingErrorPolicy="error"
Key: DAFFODIL-2861
URL: https://issues.apache.org/jira/browse/DAFFODIL-2861
Project: Daffodil
Issue Type: Bug
Components: Back End
Affects Versions: 3.0.0
Reporter: Steve Lawrence
If a schema sets {{dfdl:encodingErrorPolicy="error"}}, then Daffodil outputs
this warning:
> [warning] Schema Definition Warning: dfdl:encodingErrorPolicy="error" is not
> yet implemented. The 'replace' value will be used.
The idea is we want to be portable with schemas written for IBM DFDL (which
only supports "error" and not "replace"), so we support the "error" property
value but use the our only supported "replace" behavior. In the common case
where there are no encoding errors, the behavior will be exactly the same as
IBM DFDL.
But if there is an encoding error, then Daffodil throws this assertion:
{code}
!!
!! Not Yet Implemented !!
!!
You are using a feature that is not yet implemented:
Some(Not yet implemented: dfdl:encodingErrorPolicy="error"
org.apache.daffodil.lib.exceptions.Assert$.nyi(Assert.scala:151)
org.apache.daffodil.io.processors.charset.BitsCharsetDecoder.decodeOneHandleMalformed(BitsCharsetDecoder.scala:122)
org.apache.daffodil.io.processors.charset.BitsCharsetDecoder.decode(BitsCharsetDecoder.scala:84)
org.apache.daffodil.io.InputSourceDataInputStreamCharIterator.fetch(InputSourceDataInputStream.scala:906))
{code}
So even though the warning says we are going to use "replace", we still clearly
use "error", which results in an assertion failure that we don't support it.
I confirmed this occurs at least as far back as Daffodil 3.0.0.
Here is a schema that reproduces the issue:
{code:xml}
http://www.ogf.org/dfdl/;>
{code}
Any file that non-ASCII characters followed by a new line should trigger this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DAFFODIL-2861) Assertion thrown when encoding error with dfdl:encodingErrorPolicy="error"
[
https://issues.apache.org/jira/browse/DAFFODIL-2861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence updated DAFFODIL-2861:
-
Description:
If a schema sets {{dfdl:encodingErrorPolicy="error"}}, then Daffodil outputs
this warning:
{quote}
[warning] Schema Definition Warning: dfdl:encodingErrorPolicy="error" is not
yet implemented. The 'replace' value will be used.
{quote}
The idea is we want to be portable with schemas written for IBM DFDL (which
only supports "error" and not "replace"), so we support the "error" property
value but use the our only supported "replace" behavior. In the common case
where there are no encoding errors, the behavior will be exactly the same as
IBM DFDL.
But if there is an encoding error, then Daffodil throws this assertion:
{code}
!!
!! Not Yet Implemented !!
!!
You are using a feature that is not yet implemented:
Some(Not yet implemented: dfdl:encodingErrorPolicy="error"
org.apache.daffodil.lib.exceptions.Assert$.nyi(Assert.scala:151)
org.apache.daffodil.io.processors.charset.BitsCharsetDecoder.decodeOneHandleMalformed(BitsCharsetDecoder.scala:122)
org.apache.daffodil.io.processors.charset.BitsCharsetDecoder.decode(BitsCharsetDecoder.scala:84)
org.apache.daffodil.io.InputSourceDataInputStreamCharIterator.fetch(InputSourceDataInputStream.scala:906))
{code}
So even though the warning says we are going to use "replace", we still clearly
use "error", which results in an assertion failure that we don't support it.
I confirmed this occurs at least as far back as Daffodil 3.0.0.
Here is a schema that reproduces the issue:
{code:xml}
http://www.ogf.org/dfdl/;>
{code}
Any file that non-ASCII characters followed by a new line should trigger this.
was:
If a schema sets {{dfdl:encodingErrorPolicy="error"}}, then Daffodil outputs
this warning:
> [warning] Schema Definition Warning: dfdl:encodingErrorPolicy="error" is not
> yet implemented. The 'replace' value will be used.
The idea is we want to be portable with schemas written for IBM DFDL (which
only supports "error" and not "replace"), so we support the "error" property
value but use the our only supported "replace" behavior. In the common case
where there are no encoding errors, the behavior will be exactly the same as
IBM DFDL.
But if there is an encoding error, then Daffodil throws this assertion:
{code}
!!
!! Not Yet Implemented !!
!!
You are using a feature that is not yet implemented:
Some(Not yet implemented: dfdl:encodingErrorPolicy="error"
org.apache.daffodil.lib.exceptions.Assert$.nyi(Assert.scala:151)
org.apache.daffodil.io.processors.charset.BitsCharsetDecoder.decodeOneHandleMalformed(BitsCharsetDecoder.scala:122)
org.apache.daffodil.io.processors.charset.BitsCharsetDecoder.decode(BitsCharsetDecoder.scala:84)
org.apache.daffodil.io.InputSourceDataInputStreamCharIterator.fetch(InputSourceDataInputStream.scala:906))
{code}
So even though the warning says we are going to use "replace", we still clearly
use "error", which results in an assertion failure that we don't support it.
I confirmed this occurs at least as far back as Daffodil 3.0.0.
Here is a schema that reproduces the issue:
{code:xml}
http://www.ogf.org/dfdl/;>
{code}
Any file that non-ASCII characters followed by a new line should trigger this.
> Assertion thrown when encoding error with dfdl:encodingErrorPolicy="error"
> --
>
> Key: DAFFODIL-2861
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2861
> Project: Daffodil
> Issue Type: Bug
> Components: Back End
>Affects Versions: 3.0.0
>Reporter: Steve Lawrence
>Priority: Major
>
> If a schema sets {{dfdl:encodingErrorPolicy="error"}}, then Daffodil outputs
> this warning:
> {quote}
> [warning] Schema Definition Warning: dfdl:encodingErrorPolicy="error" is not
> yet implemented. The 'replace' value will be used.
> {quote}
> The idea is we want to be portable with schemas written for IBM DFDL (which
> only supports "error" and not "replace"), so we support the "error" property
> value but use the our only supported "replace" behavior. In the common case
> where there are no encoding errors, the behavior will be exactly the same as
> IBM DFDL.
> But if there is an encoding error, then Daffodil throws this assertion:
> {code}
> !!
> !! Not Yet Implemented !!
>
[jira] [Resolved] (DAFFODIL-2854) Release Daffodil 3.6.0
[ https://issues.apache.org/jira/browse/DAFFODIL-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2854. -- Resolution: Fixed Version bumped in commit 01accb89304637def50c6ef53bcd1adfb1f8b9e1. Released Nov 3, 2023 > Release Daffodil 3.6.0 > -- > > Key: DAFFODIL-2854 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2854 > Project: Daffodil > Issue Type: Bug >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.6.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DAFFODIL-2854) Release Daffodil 3.6.0
[ https://issues.apache.org/jira/browse/DAFFODIL-2854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence closed DAFFODIL-2854. > Release Daffodil 3.6.0 > -- > > Key: DAFFODIL-2854 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2854 > Project: Daffodil > Issue Type: Bug >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.6.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2860) ratExcludes contains files that do not exist
Steve Lawrence created DAFFODIL-2860:
Summary: ratExcludes contains files that do not exist
Key: DAFFODIL-2860
URL: https://issues.apache.org/jira/browse/DAFFODIL-2860
Project: Daffodil
Issue Type: Bug
Components: Infrastructure
Reporter: Steve Lawrence
The excludes list in project/Rat.scala contains two entries that do not exist
in the repo:
file(".idea")
file("daffodil-test/src/test/resources/org/apache/daffodil/infoset/stringAsXml/namespaced/binMessage_02.dat")
I'm not sure where the second one ended up (maybe moved, maybe deleted) but we
should remove these so our excludes list is accurate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2857) Different prefixes for same namespace fails to compile with dfdlx:repType
[
https://issues.apache.org/jira/browse/DAFFODIL-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence resolved DAFFODIL-2857.
--
Fix Version/s: 3.6.0
Resolution: Fixed
Fixed in commit d4fc818c3b27bf26abfd74663377886a8035f83c
> Different prefixes for same namespace fails to compile with dfdlx:repType
> -
>
> Key: DAFFODIL-2857
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2857
> Project: Daffodil
> Issue Type: Bug
> Components: Front End
>Reporter: Steve Lawrence
>Priority: Major
> Fix For: 3.6.0
>
>
> Say we have these two schemas:
> *foo.dfdl.xsd*
> {code:xml}
>xmlns="http://www.w3.org/2001/XMLSchema;
> xmlns:xs="http://www.w3.org/2001/XMLSchema;
> xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/;
> xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions;
> xmlns:foo="http://example.com/;
> targetNamespace="http://example.com/;
> elementFormDefault="unqualified">
>schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
>
>
> http://www.ogf.org/dfdl/;>
>
>
>
>
>
> {code}
> *bar.dfdl.xsd*
> {code:xml}
>xmlns="http://www.w3.org/2001/XMLSchema;
> xmlns:xs="http://www.w3.org/2001/XMLSchema;
> xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/;
> xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions;
> xmlns:bar="http://example.com/;
> targetNamespace="http://example.com/;
> elementFormDefault="unqualified">
>schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
>
> http://www.ogf.org/dfdl/;>
>
>
>
>
>
>
>
>
>
>
>
>
>
> {code}
> Note that foo.dfdl.xsd and bar.dfdl.xsd use the same targetNamespace, but
> have different namespace prefixes.
> Trying to compile this schema causes the error:
> > [error] Schema Definition Error: Undefined QName prefix 'bar'
> Likely what is happening is we create a quasi element in the scope
> foo.dfdl.xsd that has its type set to that of its dfdlx:repType property,
> which is "bar:enum". But in that scope, "bar:enum" doesn't make sense because
> the "bar" prefix isn't defined, the namespace prefix in this scope is "foo".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2858) Whitepace not allowed around DFDL List of String Literal
[ https://issues.apache.org/jira/browse/DAFFODIL-2858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2858. -- Resolution: Fixed Fixed in commit b5d0ec7278cfe42dcf706ba4970749b065de6f8d > Whitepace not allowed around DFDL List of String Literal > > > Key: DAFFODIL-2858 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2858 > Project: Daffodil > Issue Type: Bug > Components: Front End >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.6.0 > > > We currently SDE if there is whitespace around a property that is a List of > DFDL String Literal. We already allow for collapsing internal whitespace in > for these properties. We should remove this restriction and aslo allow > removing leading and trailing whitespace. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2858) Whitepace not allowed around DFDL List of String Literal
Steve Lawrence created DAFFODIL-2858: Summary: Whitepace not allowed around DFDL List of String Literal Key: DAFFODIL-2858 URL: https://issues.apache.org/jira/browse/DAFFODIL-2858 Project: Daffodil Issue Type: Bug Components: Front End Reporter: Steve Lawrence Assignee: Steve Lawrence Fix For: 3.6.0 We currently SDE if there is whitespace around a property that is a List of DFDL String Literal. We already allow for collapsing internal whitespace in for these properties. We should remove this restriction and aslo allow removing leading and trailing whitespace. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2857) Different prefixes for same namespace fails to compile wih dfdlx:repType
Steve Lawrence created DAFFODIL-2857:
Summary: Different prefixes for same namespace fails to compile
wih dfdlx:repType
Key: DAFFODIL-2857
URL: https://issues.apache.org/jira/browse/DAFFODIL-2857
Project: Daffodil
Issue Type: Bug
Components: Front End
Reporter: Steve Lawrence
Say we have these two schemas:
*foo.dfdl.xsd*
{code:xml}
http://www.w3.org/2001/XMLSchema;
xmlns:xs="http://www.w3.org/2001/XMLSchema;
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/;
xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions;
xmlns:foo="http://example.com/;
targetNamespace="http://example.com/;
elementFormDefault="unqualified">
http://www.ogf.org/dfdl/;>
{code}
*bar.dfdl.xsd*
{code:xml}
http://www.w3.org/2001/XMLSchema;
xmlns:xs="http://www.w3.org/2001/XMLSchema;
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/;
xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions;
xmlns:bar="http://example.com/;
targetNamespace="http://example.com/;
elementFormDefault="unqualified">
http://www.ogf.org/dfdl/;>
{code}
Note that foo.dfdl.xsd and bar.dfdl.xsd use the same targetNamespace, but have
different namespace prefixes.
Trying to compile this schema causes the error:
> [error] Schema Definition Error: Undefined QName prefix 'bar'
Likely what is happening is we create a quasi element in the scope foo.dfdl.xsd
that has its type set to that of its dfdlx:repType property, which is
"bar:enum". But in that scope, "bar:enum" doesn't make sense because the "bar"
prefix isn't defined, the namespace prefix in this scope is "foo".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DAFFODIL-2857) Different prefixes for same namespace fails to compile with dfdlx:repType
[
https://issues.apache.org/jira/browse/DAFFODIL-2857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence updated DAFFODIL-2857:
-
Summary: Different prefixes for same namespace fails to compile with
dfdlx:repType (was: Different prefixes for same namespace fails to compile wih
dfdlx:repType)
> Different prefixes for same namespace fails to compile with dfdlx:repType
> -
>
> Key: DAFFODIL-2857
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2857
> Project: Daffodil
> Issue Type: Bug
> Components: Front End
>Reporter: Steve Lawrence
>Priority: Major
>
> Say we have these two schemas:
> *foo.dfdl.xsd*
> {code:xml}
>xmlns="http://www.w3.org/2001/XMLSchema;
> xmlns:xs="http://www.w3.org/2001/XMLSchema;
> xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/;
> xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions;
> xmlns:foo="http://example.com/;
> targetNamespace="http://example.com/;
> elementFormDefault="unqualified">
>schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
>
>
> http://www.ogf.org/dfdl/;>
>
>
>
>
>
> {code}
> *bar.dfdl.xsd*
> {code:xml}
>xmlns="http://www.w3.org/2001/XMLSchema;
> xmlns:xs="http://www.w3.org/2001/XMLSchema;
> xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/;
> xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions;
> xmlns:bar="http://example.com/;
> targetNamespace="http://example.com/;
> elementFormDefault="unqualified">
>schemaLocation="/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
>
> http://www.ogf.org/dfdl/;>
>
>
>
>
>
>
>
>
>
>
>
>
>
> {code}
> Note that foo.dfdl.xsd and bar.dfdl.xsd use the same targetNamespace, but
> have different namespace prefixes.
> Trying to compile this schema causes the error:
> > [error] Schema Definition Error: Undefined QName prefix 'bar'
> Likely what is happening is we create a quasi element in the scope
> foo.dfdl.xsd that has its type set to that of its dfdlx:repType property,
> which is "bar:enum". But in that scope, "bar:enum" doesn't make sense because
> the "bar" prefix isn't defined, the namespace prefix in this scope is "foo".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DAFFODIL-2856) Configure SonarCloud scan run on Java 17
Steve Lawrence created DAFFODIL-2856:
Summary: Configure SonarCloud scan run on Java 17
Key: DAFFODIL-2856
URL: https://issues.apache.org/jira/browse/DAFFODIL-2856
Project: Daffodil
Issue Type: Bug
Reporter: Steve Lawrence
Received an email with the following from SonarCloud:
{quote}
Starting from the 15th of November 2023, SonarCloud will no longer accept scans
executed using Java 11. This decision has been made to ensure the platform's
compatibility with the latest technologies and to provide you with an enhanced
experience.
To continue benefiting from SonarCloud's advanced features and analysis
capabilities, we strongly encourage you to migrate your runtime environment for
the analysis to Java 17 as soon as possible.{quote}
We should update our github action to run on our Java 17 matrix instead of Java
11:
https://github.com/apache/daffodil/blob/main/.github/workflows/main.yml#L88-L95
Marking as critical since this will happen fairly soon, and we don't want
builds to start breaking.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DAFFODIL-2856) Configure SonarCloud scan run on Java 17
[
https://issues.apache.org/jira/browse/DAFFODIL-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Lawrence updated DAFFODIL-2856:
-
Component/s: Infrastructure
> Configure SonarCloud scan run on Java 17
>
>
> Key: DAFFODIL-2856
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2856
> Project: Daffodil
> Issue Type: Bug
> Components: Infrastructure
>Reporter: Steve Lawrence
>Priority: Critical
>
> Received an email with the following from SonarCloud:
> {quote}
> Starting from the 15th of November 2023, SonarCloud will no longer accept
> scans executed using Java 11. This decision has been made to ensure the
> platform's compatibility with the latest technologies and to provide you with
> an enhanced experience.
> To continue benefiting from SonarCloud's advanced features and analysis
> capabilities, we strongly encourage you to migrate your runtime environment
> for the analysis to Java 17 as soon as possible.{quote}
> We should update our github action to run on our Java 17 matrix instead of
> Java 11:
> https://github.com/apache/daffodil/blob/main/.github/workflows/main.yml#L88-L95
> Marking as critical since this will happen fairly soon, and we don't want
> builds to start breaking.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (DAFFODIL-2855) Scoping different when using dfldx:repType vs type
Steve Lawrence created DAFFODIL-2855:
Summary: Scoping different when using dfldx:repType vs type
Key: DAFFODIL-2855
URL: https://issues.apache.org/jira/browse/DAFFODIL-2855
Project: Daffodil
Issue Type: Bug
Components: Front End
Reporter: Steve Lawrence
Fix For: 4.0.0
It looks like our property scoping rules may be broken, and have been so for a
while.
Assume we have these two schemas:
**foo.dfdl.xsd**
{code:xml}
http://www.w3.org/2001/XMLSchema;
xmlns:xs="http://www.w3.org/2001/XMLSchema;
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/;
xmlns:dfdlx="http://www.ogf.org/dfdl/dfdl-1.0/extensions;
xmlns:ex="http://example.com;
targetNamespace="http://example.com;
elementFormDefault="unqualified">
http://www.ogf.org/dfdl/;>
{code}
**bar.dfdl.xsd**
{code:xml}
http://www.w3.org/2001/XMLSchema;
xmlns:xs="http://www.w3.org/2001/XMLSchema;
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/;
xmlns:ex="http://example.com;
targetNamespace="http://example.com;
elementFormDefault="unqualified">
http://www.ogf.org/dfdl/;>
{code}
Note that foo.dfdl.xsd defines the byteOrder to be littleEndian, whereas
bar.dfdl.xsd defines the byteOrer to be bigEndian. According to scoping rules,
the uint16 type is a binary, bigEndian, 2-byte short.
And it assumes the data to be parsed is 0x00 0xFF, which in bigEndian is 255
and in littleEndian is 65280.
Both the foo1 and foo2 ultimately have the same representation type of "uint16"
(foo1 via a simpleType with the dfdlx:repType=uint16, and foo2 with the
type=uint16) so we would expect similar results.
However, with Daffodil 3.6.0, foo1 parses 0x00FF as bigEndian (255) and foo2
parses as little endian (65820). After some discussions, we believe bigEndian
is the correct behavior, since uint16 is defined in bar.dfdl.xsd with a
bigEndian default format.
Also note that with Daffodil 3.5.0, both foo1 and foo2 parsed as the incorrect
(but consistent) littleEndian. Changes to repType in 3.6.0 seemed to have
changed only scoping for repType.
So all this seems to mean that normal property lookups are broken and have
probably been broken for a long time since we haven't changed scoping rules in
a very long time. dfdlx:repType lookups appear to behave as expected in 3.6.0,
but were incorrect in 3.5.0. It's possible some schemas relied on that broken
behavior and may need to be updated to work with 3.6.0
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (DAFFODIL-2829) Prepare for 3.6.0 development
[ https://issues.apache.org/jira/browse/DAFFODIL-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence updated DAFFODIL-2829: - Summary: Prepare for 3.6.0 development (was: Prepare for 3.6.0 relese) > Prepare for 3.6.0 development > - > > Key: DAFFODIL-2829 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2829 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.6.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (DAFFODIL-2829) Prepare for 3.6.0 development
[ https://issues.apache.org/jira/browse/DAFFODIL-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence closed DAFFODIL-2829. > Prepare for 3.6.0 development > - > > Key: DAFFODIL-2829 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2829 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.6.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2829) Prepare for 3.6.0 development
[ https://issues.apache.org/jira/browse/DAFFODIL-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2829. -- Resolution: Fixed > Prepare for 3.6.0 development > - > > Key: DAFFODIL-2829 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2829 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.6.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (DAFFODIL-2829) Prepare for 3.6.0 relese
[ https://issues.apache.org/jira/browse/DAFFODIL-2829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence reopened DAFFODIL-2829: -- > Prepare for 3.6.0 relese > > > Key: DAFFODIL-2829 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2829 > Project: Daffodil > Issue Type: Bug > Components: Infrastructure >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.6.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2854) Release Daffodil 3.6.0
Steve Lawrence created DAFFODIL-2854: Summary: Release Daffodil 3.6.0 Key: DAFFODIL-2854 URL: https://issues.apache.org/jira/browse/DAFFODIL-2854 Project: Daffodil Issue Type: Bug Reporter: Steve Lawrence Assignee: Steve Lawrence Fix For: 3.6.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (DAFFODIL-2852) VariableMap copies during suspensions cause performance degredation
[ https://issues.apache.org/jira/browse/DAFFODIL-2852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Lawrence resolved DAFFODIL-2852. -- Resolution: Fixed Fixed in commit f3fb555c98aa3cfe24270f38b579b03316cc8ad0 > VariableMap copies during suspensions cause performance degredation > --- > > Key: DAFFODIL-2852 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2852 > Project: Daffodil > Issue Type: Bug > Components: Back End, Unparsing >Reporter: Steve Lawrence >Assignee: Steve Lawrence >Priority: Major > Fix For: 3.6.0 > > > When a suspension is created when unparsing, we create a copy of the > VariableMap so that future newVariableInstances do not affect the supension. > The current VariableMap internals uses a Map of ArrayBuffers, both of which > are fairly expensive to copy, with the overhead showing up heavily when > profiling. We should change the VariableMap internals to use a data structure > that allows fast copies, while still maintaining constant time lookups. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2852) VariableMap copies during suspensions cause performance degredation
Steve Lawrence created DAFFODIL-2852: Summary: VariableMap copies during suspensions cause performance degredation Key: DAFFODIL-2852 URL: https://issues.apache.org/jira/browse/DAFFODIL-2852 Project: Daffodil Issue Type: Bug Components: Back End, Unparsing Reporter: Steve Lawrence Assignee: Steve Lawrence Fix For: 3.6.0 When a suspension is created when unparsing, we create a copy of the VariableMap so that future newVariableInstances do not affect the supension. The current VariableMap internals uses a Map of ArrayBuffers, both of which are fairly expensive to copy, with the overhead showing up heavily when profiling. We should change the VariableMap internals to use a data structure that allows fast copies, while still maintaining constant time lookups. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (DAFFODIL-2851) Excessive alloations in StringOfSpecifiedLengthMixin
[ https://issues.apache.org/jira/browse/DAFFODIL-2851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776188#comment-17776188 ] Steve Lawrence commented on DAFFODIL-2851: -- As evidence that this is a real problem, I reduced the "maximumSimpleElementSizeInCharacters" tunable from 1048576 to 50 and saw an increase in performance of about 20%+. Granted the file I'm am testing is sort of a perfect case for this improvement--it is a very small file (66 bytes) and that file is all integers except for a single 7 character string. So parsing this file allocates a huge char buffer once per parse and never uses it again. In a more complex file with many more strings, this buffer would be reused. But I imagine we still would rarely need a 1MB char buffer. With this change, this char buffer allocation disappears from profiling, and is just amongst the noise. > Excessive alloations in StringOfSpecifiedLengthMixin > > > Key: DAFFODIL-2851 > URL: https://issues.apache.org/jira/browse/DAFFODIL-2851 > Project: Daffodil > Issue Type: Bug > Components: Back End >Reporter: Steve Lawrence >Priority: Major > > The StringOfSpecifiedLengthMixin passes in the value of the > "maximumSimpleElementSizeInCharacters" tunable to the getSomeString function: > https://github.com/apache/daffodil/blob/main/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/parsers/StringLengthParsers.scala#L89-L94 > The getSomeString function calls withLocalCharBuffer which allocates a char > buffer of that size where it will decode the string. Currently, the tunable > defaults to 1MB. This size is pretty large, large enough to be a noticeable > contributor to allocations and cpu usage when profiling. > Fortunately, the allocated char buffer is cached and reused during the parse > (though each parse allocates a new one), so it's only a one time penalty per > parse. But most files are not going to have single strings nearly that large > so this large allocation is just a waste. > We should consider ways to reduce this allocation. Maybe simply decrease the > tunable? Or maybe change the logic so StringOfSpecifiedLength allocates a > much smaller amount, and grows the buffer if needed, maybe taking into > account bitLimit? Or maybe the buffer is shared among different parses in a > ThreadLocal, so we still allocate a large buffer, but the penalty is only > once per thread instead of once per parse? Likely other options... -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (DAFFODIL-2851) Excessive alloations in StringOfSpecifiedLengthMixin
Steve Lawrence created DAFFODIL-2851: Summary: Excessive alloations in StringOfSpecifiedLengthMixin Key: DAFFODIL-2851 URL: https://issues.apache.org/jira/browse/DAFFODIL-2851 Project: Daffodil Issue Type: Bug Components: Back End Reporter: Steve Lawrence The StringOfSpecifiedLengthMixin passes in the value of the "maximumSimpleElementSizeInCharacters" tunable to the getSomeString function: https://github.com/apache/daffodil/blob/main/daffodil-runtime1/src/main/scala/org/apache/daffodil/runtime1/processors/parsers/StringLengthParsers.scala#L89-L94 The getSomeString function calls withLocalCharBuffer which allocates a char buffer of that size where it will decode the string. Currently, the tunable defaults to 1MB. This size is pretty large, large enough to be a noticeable contributor to allocations and cpu usage when profiling. Fortunately, the allocated char buffer is cached and reused during the parse (though each parse allocates a new one), so it's only a one time penalty per parse. But most files are not going to have single strings nearly that large so this large allocation is just a waste. We should consider ways to reduce this allocation. Maybe simply decrease the tunable? Or maybe change the logic so StringOfSpecifiedLength allocates a much smaller amount, and grows the buffer if needed, maybe taking into account bitLimit? Or maybe the buffer is shared among different parses in a ThreadLocal, so we still allocate a large buffer, but the penalty is only once per thread instead of once per parse? Likely other options... -- This message was sent by Atlassian Jira (v8.20.10#820010)
