GitHub user stevedlawrence opened a pull request:
https://github.com/apache/incubator-daffodil/pull/13
Fix truncation bug when unparsing escapeKind="character"
Daffodil had a logic bug when unparsing a field with an escape scheme
with escapeKind="character". In such a case, we run the field DFA
looking at each character to unparse to see if it could potentially
start a delimiter and thus may need to be escaped. If we do find such a
character, we then run each delimiter DFA on the input to determine if
the delimiter actually existed or if we just found a character the looks
like it starts a delimiter but actually doesn't. We then gather up
successful matches for a longest match check. This logic is all
correct.
The problem is that for escapeKind="character", each time a delimiter
DFA failed to match, we would move the field register to the next rule. So
if we checked for 10 delimiters and 9 failed, we would advance the field
register 9 times, which is wrong and would cause the field register to
end prematurely, leading to truncated unparsed data.
Instead, we should wait until all the delimiter DFAs are checked, and
only if none match (i.e. success.isEmpty is true) then we advance the
field register by one rule to add that character to the field. This
patch fixes the text delimited unparser to use this logic for
escapeKind="character". This is actually very similar logic that
escapeKind="block" had, so this matches that correct behavior. Comments
were also added to clarify this section of code.
DAFFODIL-1851
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/stevedlawrence/incubator-daffodil
daffodil-1851-escape-scheme
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-daffodil/pull/13.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13
----
commit 066bdecd33f70b19a09ef3970b54265b84e308a0
Author: Steve Lawrence <[email protected]>
Date: 2017-11-30T13:02:33Z
Fix truncation bug when unparsing escapeKind="character"
Daffodil had a logic bug when unparsing a field with an escape scheme
with escapeKind="character". In such a case, we run the field DFA
looking at each character to unparse to see if it could potentially
start a delimiter and thus may need to be escaped. If we do find such a
character, we then run each delimiter DFA on the input to determine if
the delimiter actually existed or if we just found a character the looks
like it starts a delimiter but actually doesn't. We then gather up
successful matches for a longest match check. This logic is all
correct.
The problem is that for escapeKind="character", each time a delimiter
DFA failed to match, we would move the field register to the next rule. So
if we checked for 10 delimiters and 9 failed, we would advance the field
register 9 times, which is wrong and would cause the field register to
end prematurely, leading to truncated unparsed data.
Instead, we should wait until all the delimiter DFAs are checked, and
only if none match (i.e. success.isEmpty is true) then we advance the
field register by one rule to add that character to the field. This
patch fixes the text delimited unparser to use this logic for
escapeKind="character". This is actually very similar logic that
escapeKind="block" had, so this matches that correct behavior. Comments
were also added to clarify this section of code.
DAFFODIL-1851
----
---
