[
https://issues.apache.org/jira/browse/DAFFODIL-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16622211#comment-16622211
]
Steve Lawrence commented on DAFFODIL-2007:
------------------------------------------
Two issues were found in the VMF schemas:
1. There are a couple of non-ascii characters (e.g non-breaking space) in the
VMF schemas that may have caused encoding issues on windows. This seems a bit
odd since the non-ascii characters are completely valid UTF-8, but we have seen
cases where certain UTF-8 files caused issues many years ago when we did test
on windows. These characters are not necessary and have been removed.
2. Windows git tries to be helpful and replaces all unix style line endings in
the source with windows style line endings, and converts them back when making
a commit. However, our schema generator explicitly uses unix style line endings
except for a couple of places. So on windows it's very easy to end up with
files with mix of unix and windows style line endings. I think this may have
also caused some encoding issues. I've modified the schema generator to ensure
unix style line endings are always used when generating schemas.
With these changes to the VMF schema, things appear to work with Daffodil 2.2.0
on windows.
We probably still want some changes to Daffodil to improve the diagnostics here
to mention there's some encoding issue.
> VMF fails to run tests on MS-Windows
> ------------------------------------
>
> Key: DAFFODIL-2007
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2007
> Project: Daffodil
> Issue Type: Bug
> Components: Windows
> Affects Versions: 2.2.0
> Reporter: Michael Beckerle
> Assignee: Steve Lawrence
> Priority: Major
>
> Users trying to test VMF on windows report that 'sbt test' works fine on
> mil-std-2045, but fails for VMF.
> Some debugging shows that when the DaffodilConstructingLoader creates a
> scala.io.Source from the URI, the resulting source faults on the first
> hasNext call.
> I added some instrumentation to DaffodilConstructingLoader.scala:
> {{class DaffodilConstructingLoader(uri: URI, errorHandler:
> org.xml.sax.ErrorHandler)
> extends ConstructingParser({
> //
> // Note: we must open the XML carefully since it might be in some non
> // default encoding (we have tests that have UTF-16 for example)
> //
> val is = uri.toURL.openStream()
> System.err.println("URI to open: " + uri)
> val rdr = new XmlStreamReader(is) // apache has a good thing for
> determining XML charset
> val csName = rdr.getEncoding
> System.err.println("Charset is: " + csName)
> import collection.JavaConverters._
> val lines = IOUtils.readLines(rdr).asScala
> System.err.println("First line is " +
> lines.headOption.getOrElse("Nothing."))
> rdr.close()
> val source = Source.fromURL(uri.toURL, csName) // tbd: can one create a
> Source somehow directly from the rdr?
> val hn = source.hasNext
> System.err.println("First hasNext call succeeded with result: " + hn)
> source
> }, true) {}}
> The resulting print out for running the single test
> TestVMFFullMessages.test_K00_1 is below. It never gets to the last println in
> the above code. It fails on the hasNext call on the Source. Prior print
> statements show the file can be opened and accessed, so it's something to do
> with the scala.io.Source, and creating one from a URL that is the problem it
> seems.
> Here's the error:
> {{URI to open:
> file:/E:/dataiti/git/fouo-schemas/vmf/schema/target/classes/com/tresys/mil-std-6017/xsd/vmf.main.gen.dfdl.xsd
> Charset is: UTF-8
> First line is <?xml version='1.0' encoding='UTF-8'?>
> [error] Test com.tresys.vmf.TestVMFFullMessages.test_K00_1 failed:
> java.nio.charset.MalformedInputException: Input length = 1, took 4.901 sec
> [error] at
> java.nio.charset.CoderResult.throwException(CoderResult.java:281)
> [error] at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:339)
> [error] at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> [error] at java.io.InputStreamReader.read(InputStreamReader.java:184)
> [error] at java.io.BufferedReader.fill(BufferedReader.java:161)
> [error] at java.io.BufferedReader.read(BufferedReader.java:182)
> [error] at
> scala.io.BufferedSource.$anonfun$iter$2(BufferedSource.scala:38)
> [error] at scala.io.Codec.wrap(Codec.scala:69)
> [error] at
> scala.io.BufferedSource.$anonfun$iter$1(BufferedSource.scala:38)
> [error] at
> scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:12)
> [error] at scala.collection.Iterator$$anon$9.next(Iterator.scala:170)
> [error] at scala.collection.Iterator$$anon$15.hasNext(Iterator.scala:652)
> [error] at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:456)
> [error] at scala.io.Source.hasNext(Source.scala:246)
> [error] at scala.xml.parsing.MarkupParser.ch(MarkupParser.scala:103)
> [error] at scala.xml.parsing.MarkupParser.ch$(MarkupParser.scala:101)
> [error] at
> scala.xml.parsing.ConstructingParser.ch(ConstructingParser.scala:48)
> [error] at scala.xml.parsing.MarkupParser.eof(MarkupParser.scala:129)
> [error] at scala.xml.parsing.MarkupParser.eof$(MarkupParser.scala:129)
> [error] at
> scala.xml.parsing.ConstructingParser.eof(ConstructingParser.scala:48)
> [error] at scala.xml.parsing.MarkupParser.xComment(MarkupParser.scala:392)
> [error] at
> scala.xml.parsing.MarkupParser.xComment$(MarkupParser.scala:389)
> [error] at
> scala.xml.parsing.ConstructingParser.xComment(ConstructingParser.scala:48)
> [error] at scala.xml.parsing.MarkupParser.content1(MarkupParser.scala:428)
> [error] at
> scala.xml.parsing.MarkupParser.content1$(MarkupParser.scala:419)
> [error] at
> scala.xml.parsing.ConstructingParser.content1(ConstructingParser.scala:48)
> [error] at scala.xml.parsing.MarkupParser.content(MarkupParser.scala:459)
> [error] at scala.xml.parsing.MarkupParser.content$(MarkupParser.scala:442)
> [error] at
> scala.xml.parsing.ConstructingParser.content(ConstructingParser.scala:48)
> [error] at scala.xml.parsing.MarkupParser.document(MarkupParser.scala:244)
> [error] at
> scala.xml.parsing.MarkupParser.document$(MarkupParser.scala:225)
> [error] at
> scala.xml.parsing.ConstructingParser.document(ConstructingParser.scala:48)
> [error] at
> org.apache.daffodil.xml.DaffodilConstructingLoader.load(DaffodilConstructingLoader.scala:178)
> [error] at
> org.apache.daffodil.xml.DaffodilXMLLoader.load(DaffodilXMLLoader.scala:530)}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
