Will do.
Thanks a lot.
Meanwhile Mike, will I still be able to use Apache Daffodil for the following
conversions as well?
* Convert XML or JSON back to fixed length text file.
* Convert XML file to JSON file.
* Convert JSON file to XML file.
From: Mike Beckerle <mbecke...@apache.org>
Sent: Tuesday, April 5, 2022 3:05 PM
To: users@daffodil.apache.org
Subject: [**EXTERNAL**] Re: Request help with creating annotations for fixed
length to XML and vice versa and JSON to XML and vice versa
Be aware this external email contains an attachment and/or link.
Ensure the email and contents are expected. If there are concerns, please
submit suspicious messages to the Cyber Intelligence Center using the Report
Phishing button.
Venkataraman,
I have made a few changes to your schema, and it doesn't parse the data fully
yet, but I think I got it to where you can more easily make progress.
Suggestions are these:
1) factor out redundant properties into reusable named formats so that you
don't have so many properties cluttering the schema
2) remove NL separator from records since the data doesn't seem to have them.
3) nilled elements - these are somewhat awkward in DFDL. You can't put the
nil-oriented properties like dfdl:nilKind or dfdl:nilValue on a simpleType,
only on an element. This limits the ability to hide them nicely when reusing.
4) the nillable="true" is an XML Schema thing, not DFDL property, so that also
has to appear only on elements.
5) make a file with just one record in it and test that first.
I can now get a trace of the parse via:
daffodil -t parse -s bsc.dfdl.xsd bsc5_one_record.dat
It fails, but you can see it is making its way through the fields one by one.
So you should be able to figure out which field(s) length is incorrect and make
progress.
Good luck.
parser: <Element name='file'><Sequence>...</Sequence></Element>
bitPosition: 0
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000000: 2033 3920 3838 4761 6d20 5065 6742 442b ␣39␣88Gam␣PegBD+
00000010: 3134 2020 2031 3420 2020 2038 3836 2039 14␣␣␣14␣␣␣␣886␣9
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<ex:file xmlns:ex="http://example.com";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";></ex:file>
diff:
(no differences)
----------------------------------------------------------------- 1
parser: <Element
name='record'><Sequence>...............................................................................................................................................................</Sequence></Element>
bitPosition: 0
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000000: 2033 3920 3838 4761 6d20 5065 6742 442b ␣39␣88Gam␣PegBD+
00000010: 3134 2020 2031 3420 2020 2038 3836 2039 14␣␣␣14␣␣␣␣886␣9
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<ex:file xmlns:ex="http://example.com";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";>
<record></record>
</ex:file>
diff:
(no differences)
----------------------------------------------------------------- 2
parser: <Element
name='HR'><ConvertTextCombinatorParser>......</ConvertTextCombinatorParser></Element>
bitPosition: 0
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000000: 2033 3920 3838 4761 6d20 5065 6742 442b ␣39␣88Gam␣PegBD+
00000010: 3134 2020 2031 3420 2020 2038 3836 2039 14␣␣␣14␣␣␣␣886␣9
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR></HR>
</record>
diff:
(no differences)
----------------------------------------------------------------- 3
parser: <HR parser='StringOfSpecifiedLengthParser' />
bitPosition: 32
data:
├───────┤ ├──┤
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000000: 3838 4761 6d20 5065 6742 442b 3134 2020 88Gam␣PegBD+14␣␣
00000010: 2031 3420 2020 2038 3836 2039 3137 3831 ␣14␣␣␣␣886␣91781
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR></HR>
</record>
diff:
bitLimit: (no value) -> 32
bitPosition: 0 -> 32
----------------------------------------------------------------- 4
parser: <ConvertTextNumberParser/>
bitPosition: 32
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000000: 2033 3920 3838 4761 6d20 5065 6742 442b ␣39␣88Gam␣PegBD+
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
</record>
diff:
bitLimit: 32 -> (no value)
----------------------------------------------------------------- 5
parser: <Element name='Name'><SpecifiedLengthExplicitParser><Name
parser='StringOfSpecifiedLengthParser'
/></SpecifiedLengthExplicitParser></Element>
bitPosition: 32
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000000: 2033 3920 3838 4761 6d20 5065 6742 442b ␣39␣88Gam␣PegBD+
00000010: 3134 2020 2031 3420 2020 2038 3836 2039 14␣␣␣14␣␣␣␣886␣9
00000020: 3137 3831 2020 2037 4927 5720 2020 2020 1781␣␣␣7I'W␣␣␣␣␣
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name></Name>
</record>
diff:
childIndex: 1 -> 2
groupIndex: 1 -> 2
----------------------------------------------------------------- 6
parser: <Name parser='StringOfSpecifiedLengthParser' />
bitPosition: 112
data:
├──────────────────────┤ ├────────┤
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000000: 5065 6742 442b 3134 2020 2031 3420 2020 PegBD+14␣␣␣14␣␣␣
00000010: 2038 3836 2039 3137 3831 2020 2037 4927 ␣886␣91781␣␣␣7I'
00000020: 5720 2020 2020 2020 4761 6d20 5065 6720 W␣␣␣␣␣␣␣Gam␣Peg␣
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
</record>
diff:
bitLimit: (no value) -> 112
bitPosition: 32 -> 112
----------------------------------------------------------------- 7
parser: <Element name='DM'><SpecifiedLengthExplicitParser><DM
parser='StringOfSpecifiedLengthParser'
/></SpecifiedLengthExplicitParser></Element>
bitPosition: 112
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000000: 2033 3920 3838 4761 6d20 5065 6742 442b ␣39␣88Gam␣PegBD+
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM></DM>
</record>
diff:
bitLimit: 112 -> (no value)
childIndex: 2 -> 3
groupIndex: 2 -> 3
----------------------------------------------------------------- 8
parser: <DM parser='StringOfSpecifiedLengthParser' />
bitPosition: 200
data:
├──═ ├═
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000000: 6567 4244 2b31 3420 2020 3134 2020 2020 egBD+14␣␣␣14␣␣␣␣
00000010: 3838 3620 3931 3738 3120 2020 3749 2757 886␣91781␣␣␣7I'W
00000020: 2020 2020 2020 2047 616d 2050 6567 2020 ␣␣␣␣␣␣␣Gam␣Peg␣␣
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM>D+14 14</DM>
</record>
diff:
bitLimit: (no value) -> 200
bitPosition: 112 -> 200
----------------------------------------------------------------- 9
parser: <Element
name='HD'><SimpleNilOrValueParser>......</SimpleNilOrValueParser></Element>
bitPosition: 200
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000010: 3134 2020 2031 3420 2020 2038 3836 2039 14␣␣␣14␣␣␣␣886␣9
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM>D+14 14</DM>
<HD></HD>
</record>
diff:
bitLimit: 200 -> (no value)
childIndex: 3 -> 4
groupIndex: 3 -> 4
----------------------------------------------------------------- 10
parser: <HD nilValue='List(%ES;)'/>
bitPosition: 248
data:
├─────────────┤ ├────┤
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000010: 3420 2020 2038 3836 2039 3137 3831 2020 4␣␣␣␣886␣91781␣␣
00000020: 2037 4927 5720 2020 2020 2020 4761 6d20 ␣7I'W␣␣␣␣␣␣␣Gam␣
00000030: 5065 6720 2030 3030 3830 352e 312b 3134 Peg␣␣000805.1+14
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM>D+14 14</DM>
<HD></HD>
</record>
diff:
bitLimit: (no value) -> 248
bitPosition: 200 -> 248
failure:
Parse Error: HD - Does not contain a nil literal!
Schema context: HD Location line 61 column 18 in
file:/home/mbeckerle/Downloads/v7/bsc.dfdl.xsd
Data location was preceding byte 31 limit(bytes) 31
----------------------------------------------------------------- 11
parser: <HD parser='StringOfSpecifiedLengthParser' />
bitPosition: 248
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000010: 3134 2020 2031 3420 2020 2038 3836 2039 14␣␣␣14␣␣␣␣886␣9
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM>D+14 14</DM>
<HD></HD>
</record>
diff:
(no differences)
----------------------------------------------------------------- 12
parser: <ConvertTextNumberParser/>
bitPosition: 248
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000010: 3134 2020 2031 3420 2020 2038 3836 2039 14␣␣␣14␣␣␣␣886␣9
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM>D+14 14</DM>
<HD>886</HD>
</record>
diff:
bitLimit: 248 -> (no value)
----------------------------------------------------------------- 13
parser: <Element
name='SAO'><SimpleNilOrValueParser>......</SimpleNilOrValueParser></Element>
bitPosition: 248
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000010: 3134 2020 2031 3420 2020 2038 3836 2039 14␣␣␣14␣␣␣␣886␣9
00000020: 3137 3831 2020 2037 4927 5720 2020 2020 1781␣␣␣7I'W␣␣␣␣␣
00000030: 2020 4761 6d20 5065 6720 2030 3030 3830 ␣␣Gam␣Peg␣␣00080
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM>D+14 14</DM>
<HD>886</HD>
<SAO></SAO>
</record>
diff:
childIndex: 4 -> 5
groupIndex: 4 -> 5
----------------------------------------------------------------- 14
parser: <SAO nilValue='List(%ES;)'/>
bitPosition: 296
data:
├═ ║
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000010: 3420 2020 2038 3836 2039 3137 3831 2020 4␣␣␣␣886␣91781␣␣
00000020: 2037 4927 5720 2020 2020 2020 4761 6d20 ␣7I'W␣␣␣␣␣␣␣Gam␣
00000030: 5065 6720 2030 3030 3830 352e 312b 3134 Peg␣␣000805.1+14
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM>D+14 14</DM>
<HD>886</HD>
<SAO></SAO>
</record>
diff:
bitLimit: (no value) -> 296
bitPosition: 248 -> 296
failure:
Parse Error: SAO - Does not contain a nil literal!
Schema context: SAO Location line 62 column 18 in
file:/home/mbeckerle/Downloads/v7/bsc.dfdl.xsd
Data location was preceding byte 37 limit(bytes) 37
----------------------------------------------------------------- 15
parser: <SAO parser='StringOfSpecifiedLengthParser' />
bitPosition: 296
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000020: 3137 3831 2020 2037 4927 5720 2020 2020 1781␣␣␣7I'W␣␣␣␣␣
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM>D+14 14</DM>
<HD>886</HD>
<SAO></SAO>
</record>
diff:
(no differences)
----------------------------------------------------------------- 16
parser: <ConvertTextNumberParser/>
bitPosition: 296
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000020: 3137 3831 2020 2037 4927 5720 2020 2020 1781␣␣␣7I'W␣␣␣␣␣
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM>D+14 14</DM>
<HD>886</HD>
<SAO>91781</SAO>
</record>
diff:
bitLimit: 296 -> (no value)
----------------------------------------------------------------- 17
parser: <Element
name='FK5'><SimpleNilOrValueParser>......</SimpleNilOrValueParser></Element>
bitPosition: 296
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000020: 3137 3831 2020 2037 4927 5720 2020 2020 1781␣␣␣7I'W␣␣␣␣␣
00000030: 2020 4761 6d20 5065 6720 2030 3030 3830 ␣␣Gam␣Peg␣␣00080
00000040: 352e 312b 3134 3337 3430 3030 3133 3134 5.1+143740001314
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM>D+14 14</DM>
<HD>886</HD>
<SAO>91781</SAO>
<FK5></FK5>
</record>
diff:
childIndex: 5 -> 6
groupIndex: 5 -> 6
----------------------------------------------------------------- 18
parser: <FK5 nilValue='List(%ES;)'/>
bitPosition: 328
data:
├────────┤ ├──┤
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000020: 2020 2037 4927 5720 2020 2020 2020 4761 ␣␣␣7I'W␣␣␣␣␣␣␣Ga
00000030: 6d20 5065 6720 2030 3030 3830 352e 312b m␣Peg␣␣000805.1+
00000040: 3134 3337 3430 3030 3133 3134 2e32 2b31 143740001314.2+1
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM>D+14 14</DM>
<HD>886</HD>
<SAO>91781</SAO>
<FK5></FK5>
</record>
diff:
bitLimit: (no value) -> 328
bitPosition: 296 -> 328
failure:
Parse Error: FK5 - Does not contain a nil literal!
Schema context: FK5 Location line 63 column 18 in
file:/home/mbeckerle/Downloads/v7/bsc.dfdl.xsd
Data location was preceding byte 41 limit(bytes) 41
----------------------------------------------------------------- 19
parser: <FK5 parser='StringOfSpecifiedLengthParser' />
bitPosition: 328
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000020: 3137 3831 2020 2037 4927 5720 2020 2020 1781␣␣␣7I'W␣␣␣␣␣
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM>D+14 14</DM>
<HD>886</HD>
<SAO>91781</SAO>
<FK5></FK5>
</record>
diff:
(no differences)
----------------------------------------------------------------- 20
parser: <ConvertTextNumberParser/>
bitPosition: 328
data:
│ │
87654321 0011 2233 4455 6677 8899 aabb ccdd eeff 0123456789abcdef
00000020: 3137 3831 2020 2037 4927 5720 2020 2020 1781␣␣␣7I'W␣␣␣␣␣
infoset:
<?xml version="1.0" encoding="UTF-8"?>
<record>
<HR>39</HR>
<Name>88Gam PegB</Name>
<DM>D+14 14</DM>
<HD>886</HD>
<SAO>91781</SAO>
<FK5></FK5>
</record>
diff:
bitLimit: 328 -> (no value)
failure:
Parse Error: Unable to parse xs:int from text: 7I
Schema context: FK5 Location line 63 column 18 in
file:/home/mbeckerle/Downloads/v7/bsc.dfdl.xsd
Data location was preceding byte 41
----------------------------------------------------------------- 21
[error] Parse Error: Failed to populate record[1]. Cause: Parse Error: Unable
to parse xs:int from text: 7I
Schema context: FK5 Location line 63 column 18 in
file:/home/mbeckerle/Downloads/v7/bsc.dfdl.xsd
Data location was preceding byte 41
Schema context: record Location line 55 column 10 in
file:/home/mbeckerle/Downloads/v7/bsc.dfdl.xsd
Data location was preceding byte 41
[error] Parse Error: Unable to parse xs:int from text: 7I
Schema context: FK5 Location line 63 column 18 in
file:/home/mbeckerle/Downloads/v7/bsc.dfdl.xsd
Data location was preceding byte 41
On Tue, Apr 5, 2022 at 3:34 PM Mike Beckerle
<mbecke...@apache.org<mailto:mbecke...@apache.org>> wrote:
Venkataraman,
I started taking a look at this.
A few observations. The data appears to be fixed length 196 characters, and I
don't see any line-endings in the file.
So your main "file" element is a separated sequence with separator %NL;, but
that doesn't match the data file.
I'm going to clean up your schema a bit and send back to you. A few quick
things will make it much easier to work with.
On Tue, Apr 5, 2022 at 11:46 AM Sethuraman, Venkataraman
<venkataramansethura...@chevron.com<mailto:venkataramansethura...@chevron.com>>
wrote:
Here is the attachment again.
Please unzip it.
Thanks
From: Sethuraman, Venkataraman
Sent: Tuesday, April 5, 2022 10:38 AM
To: d...@daffodil.apache.org<mailto:d...@daffodil.apache.org>;
users@daffodil.apache.org<mailto:users@daffodil.apache.org>
Subject: Request help with creating annotations for fixed length to XML and
vice versa and JSON to XML and vice versa
Hi all
I’m trying to use Apache Daffodil for the following scenarios:
* Convert a fixed length text file to XML or JSON file.
* Convert XML or JSON back to fixed length text file.
* Convert XML file to JSON file.
* Convert JSON file to XML file.
Will it be possible to get some sample XSD files to implement the above
scenarios?
Also I tried to parse the attached sample files and their xsd from the url at .
I got error messages that I have attached as well.
Please let me know what corrections I need to make to the xsd for the sample to
run against daffodil version 3.3.
Thanks
Venkat
