On 12/28/2020 12:49 PM, Steve Lawrence wrote:
Welcome!
Thanks Steve. Have been watching Daffodil from afar for awhile, am glad to
finally become able to engage.
a) Yes, that's the correct DFDLGeneralFormat.dfdl.xsd. Though I'm
surprised you need that. That file is distributed in the Daffodil jars,
and when Daffodil compiles a schema it should be able to find it inside
a jar. So you shouldn't need to provide a separate copy of that file
when using Daffodil. If that's not the case, let us know--it's likely a bug.
Aha, just tested - that is correct.
I had felt the urge to add a local copy of the schema because double-checking
with XMLSpy throws an error on the schema otherwise:
=====================
File
C:\x-nps-gitlab\NetworkOptionalWarfare\RobodataDFDL\examples\csv\csv.dfdl.xsd
is valid but contains one or more warnings.
Unable to load a schema from
'org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd'.
I/O operation on file
'C:\x-nps-gitlab\NetworkOptionalWarfare\RobodataDFDL\examples\csv\org\apache\daffodil\xsd\DFDLGeneralFormat.dfdl.xsd'
failed.
Details
System Error 2: The system cannot find the file specified.
=====================
Not seeing a way to silence this XMLSpy error... but it appears to flag other
errors first so that is helpful.
Perhaps it would be nice if there was a well-defined online URI that an DFDL
engine might similarly provide (with online retrieval) via an XML catalog, that
would allow validation of the schema to proceed properly.
* <xs:include
schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
Meanwhile added Ant schema validation of the CSV schema, which passes without
giving a warning about the unavailable xs:include. This took some
configuration (not natively implemented in Apache Ant) and requires downloading
XMLSchema.xsd plus other files.
* https://ant.apache.org/manual/Tasks/schemavalidate.html
<echo message="xmlvalidate examples/csv/csv.dfdl.xsd"/>
<xmlvalidate file="examples/csv/csv.dfdl.xsd" failonerror="false" warn="true"
lenient="true"/>
<echo message="schemavalidate examples/csv/csv.dfdl.xsd"/>
<schemavalidate file="examples/csv/csv.dfdl.xsd" failonerror="true" warn="true"
lenient="false" fullchecking="false">
<schema namespace="http://www.w3.org/2001/XMLSchema";
file="validation/XMLSchema.xsd"/>
</schemavalidate>
<echo message="... schemavalidate passed"/>
b) This is the most recent version of the CSV schema. The issue you're
seeing is that the encodingErrorPolicy is set to "error" in
csv.dfdl.xsd, but Daffodil does not support this value--we only support
encodingErrorPolicy="replace".
Normally encodingErrorPolicy="error" should be a schema definition error
with a message like:
Daffodil does not support encodingErrorPolicy="error", use "replace"
instead"
But we found that there are a lot of schemas out there that already use
"error", particularly those created by IBM. (Note that IBM DFDL supports
"error" but not "replace", so we are incompatible with each other in
that regard).
Fortunately, most of the time this property doesn't actually matter--it
really only has an affect if you are parsing data that has encoding
errors, which is pretty uncommon. So because this property doesn't
usually matter, but we still want to support all these schemas that
already use "error", we decided to just ignore this property and always
treat it as if it have a value of "replace". That way we can support
schemas that use either "error" or "replace". But this does mean we are
ignoring a property when it is "error", so we output some warnings just
to make it clear that Daffodil might behave differently than the schema
author intended.
In this case, the schema author set it to "error" simple so that it
would work in both IBM DFDL and Daffodil.
So this warning can just be ignored. If you want to get rid of the
warnings and you don't care about portability with IBM DFDL, then the
easy solution is to just change encodingErrorPolicy to "replace" and the
warnings will go away.
Thanks for explanation.
Looks like this property is fully explained at
* DFDL Spec, 11.2.1 Property dfdl:encodingErrorPolicy
https://daffodil.apache.org/docs/dfdl/#_Toc54264422
Interoperability (via specification compliance) is of course helpful.
Warnings are also helpful, and should lead to fixes. Eliminating warnings is best case,
so that the presence of "expected" warnings doesn't hide the presence of
unexpected warnings.
Curious what the long-term resolution of this might be, presumably
- Apache Daffodil support for encodingErrorPolicy="error"
- Similar IBM DFDL support for encodingErrorPolicy="replace"
Again thanks for helping me understand Daffodil better.
On 12/28/20 3:16 PM, Don Brutzman wrote:
Am trying to parse CSV example after installing latest version of 3.0.0.
* Apache Daffodil Examples
https://daffodil.apache.org/examples
* DFDLSchemas / CSV
https://github.com/DFDLSchemas/CSV
* csv.dfdl.xsd
https://github.com/DFDLSchemas/CSV/blob/master/src/main/resources/com/tresys/csv/xsd/csv.dfdl.xsd
Had to go searching for
* <xs:include
schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd" />
and found a copy at
*
https://github.com/apache/incubator-daffodil/blob/master/daffodil-lib/src/main/resources/org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd
and also looked around for example data, which matches excerpt on
examples page:
*
https://raw.githubusercontent.com/DFDLSchemas/CSV/master/src/test/resources/com/tresys/csv/data/simpleCSV.csv
Then, using Ant invocation as follows,
<target name="daffodil.parse.csv" description="daffodil.apache.org
example">
<echo>daffodil parse csv</echo>
<exec executable="daffodil" dir="." vmlauncher="false">
<arg value="parse"/>
<arg value="--schema"/>
<arg value="examples/csv/csv.dfdl.xsd"/>
<arg value="--output"/>
<arg value="examples/csv/simpleCSV.parse.xml"/>
<arg value="examples/csv/simpleCSV.csv"/>
</exec>
</target>
... was able to produce attached simpleCSV.parse.xml (attached) which in
turn matches following and result on HTML page,
*
https://github.com/DFDLSchemas/CSV/blob/master/src/test/resources/com/tresys/csv/infosets/simpleCSV.xml
So that's good. However am also getting a lot of Daffodil schema warnings:
daffodil.parse.csv:
daffodil parse csv
[warning] Schema Definition Warning: dfdl:encodingErrorPolicy="error" is
not yet implemented. The 'replace' value will be used.
Schema context: title Location line 59 column 16 in
RobodataDFDL/examples/csv/csv.dfdl.xsd
[warning] Schema Definition Warning: dfdl:encodingErrorPolicy="error" is
not yet implemented. The 'replace' value will be used.
Schema context: item Location line 66 column 16 in
RobodataDFDL/examples/csv/csv.dfdl.xsd
[warning] Schema Definition Warning: dfdl:encodingErrorPolicy="error" is
not yet implemented. The 'replace' value will be used.
Schema context: sequence[1] Location line 54 column 8 in
RobodataDFDL/examples/csv/csv.dfdl.xsd
[warning] Schema Definition Warning: dfdl:encodingErrorPolicy="error" is
not yet implemented. The 'replace' value will be used.
Schema context: sequence[1] Location line 58 column 14 in
RobodataDFDL/examples/csv/csv.dfdl.xsd
[warning] Schema Definition Warning: dfdl:encodingErrorPolicy="error" is
not yet implemented. The 'replace' value will be used.
Schema context: sequence[1] Location line 65 column 14 in
RobodataDFDL/examples/csv/csv.dfdl.xsd
[warning] Schema Definition Warning: dfdl:encodingErrorPolicy="error" is
not yet implemented. The 'replace' value will be used.
Schema context: header Location line 55 column 10 in
RobodataDFDL/examples/csv/csv.dfdl.xsd
[warning] Schema Definition Warning: dfdl:encodingErrorPolicy="error" is
not yet implemented. The 'replace' value will be used.
Schema context: record Location line 63 column 10 in
RobodataDFDL/examples/csv/csv.dfdl.xsd
[warning] Schema Definition Warning: dfdl:encodingErrorPolicy="error" is
not yet implemented. The 'replace' value will be used.
Schema context: element reference ex:file Location line 36 in
RobodataDFDL/examples/csv/csv.dfdl.xsd
BUILD SUCCESSFUL (total time: 3 seconds)
Wondering please:
a. am i using the correct DFDLGeneralFormat.dfdl.xsd file for this example,
b. is there a more recent version that is suitable for version 3.0.0 ?
Thanks for all feedback.
all the best, Don
--
Don Brutzman Naval Postgraduate School, Code USW/Br brutz...@nps.edu
Watkins 270, MOVES Institute, Monterey CA 93943-5000 USA +1.831.656.2149
X3D graphics, virtual worlds, navy robotics http://faculty.nps.edu/brutzman
