Re: sbt exportJars causes "lengthLimitInBits.>=(0)" error when running "sbt test"

[Steve Lawrence]

Yeah, it looks like FileDocumentPart had this same behavior of returning 
-1 for files in jars as far back as I can find. So likely at somepoint 
we just started requiring a length. That's a bit harder to find where we 
did that, but ideally we can get rid of this -1 thing and get lengths of 
files in jars so code paths are exactly the same, regardless of jar vs file.

I created DAFFODIL-2679 for this issue.
On 3/18/22 3:00 PM, Mike Beckerle wrote:
This means when using IntelliJ IDEA, running tests, the TDML  runner is NOT
doing the jar export and then looking inside them.

Ok. I guess I believe that.

A TDML runner change must have happened between 3.0.0 and 3.1.0 creating
this vulnerability, because it doesn't happen in 3.0.0.

There are a few changes to the TDML runner in 3.1.0 so this is plausible.


On Fri, Mar 18, 2022 at 2:53 PM Steve Lawrence <slawre...@apache.org> wrote:

Hmm, I think this is just a bug in the Daffodil TDML runner related to
documentPart's with type="file":


https://github.com/apache/daffodil/blob/main/daffodil-tdml-lib/src/main/scala/org/apache/daffodil/tdml/TDMLRunner.scala#L2249-L2262

When exportJars is true, findTDMLResource for the file document part is
going be found inside a jar with the "jar" protocol. In that case, we
just say the length is -1L. So that's where our -1 is coming from, and
clearly we don't like that. Ideally we could fix this to get the length
of the resource from the jar.

So I think exportJars := true and documentPart type="file" are
incompatible with the current TDML Runner. I'll open a ticket.

- Steve

On 3/18/22 2:30 PM, Olabusayo Kilo wrote:
I was also able to recreate this error in the CSV repo by adding the
"exportJars := true" (note: false doesn't cause this error),

and updating the Daffodil version to the last commit before the RC
commit (i.e 613c2b0c077f56dbb425e7ebab1a2d154cbdbd06).

so:


exportJars :=true libraryDependencies ++=Seq(
    "org.apache.daffodil" %%"daffodil-tdml-processor" %"3.3.0-SNAPSHOT"
%"test",
---
)

So it seems to be a Daffodil error, since this doesn't happen with
3.0.0, but it does happen with 3.2.1

Note: I'm running sbt 1.5.7 (Ubuntu Java 11.0.14), but I believe Mike
tried several different versions when we first ran into this.

On 3/18/22 13:45, Mike Beckerle wrote:
I tried many things, including clean git clones of all related repos.
This
problem is a very bizarre interaction.

The fact that the tests ran flawlessly under intelliJ IDEA, but failed
running sbt from the command line

I searched for sbt bugs related to exportJars. I did find
https://github.com/sbt/sbt/issues/3119, but doesn't seem related or not
obviously so.

On Fri, Mar 18, 2022 at 1:25 PM Adams, Joshua<
jad...@owlcyberdefense.com>
wrote:

I can confirm that at least sbt clean does not resolve the issue.
Didn't
try a got clean, although I had just cloned the repo in question so I
don't
think it had gotten dirty in anyway either.

Josh

On Mar 18, 2022 1:22 PM, Steve Lawrence<slawre...@apache.org>  wrote:
That's surprising to me. All that exportJars is supposed to to do is
include or not include certains files on the class path and use
packages
jars instead.

And that assertion that is failing is beacuase the length of test data
is negative, which should never happen and I don't know how exportJars
would affect that.

It almost sounds like something got corrupted or needed to be rebuilt,
and changing exportJars allowed that to happen? Maybe there's some
conflict between using IntelliJ and sbt at the same time?

I wonder if doing an sbt clean or git clean would resolve the issue.


On 3/18/22 1:10 PM, Olabusayo Kilo wrote:
We noticed that only when running "sbt test" in schema projects with
the
exportJar setting, the error below was generated resulting in
failure of
all the tests. Note that running the same tests via IntelliJ/Junit
didn't reproduce this error, and the error was isolated to using sbt
to
run the tests.

The cause was isolated to having the "exportJars := true" setting in
build.sbt. Removing the setting removed the error, and got all tests
passing. Sending this out as an FYI in case someone else runs into
this
in the future.

[info] Test a.b.c started
[error] Test a.b.c failed: org.apache.daffodil.exceptions.Abort: Usage
error: lengthLimitInBits.>=(0)
[error] org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:137)
[error]


org.apache.daffodil.tdml.processor.DaffodilTDMLDFDLProcessor.doParseWithBothApis(DaffodilTDMLDFDLProcessor.scala:290)


[error]


org.apache.daffodil.tdml.processor.DaffodilTDMLDFDLProcessor.parse(DaffodilTDMLDFDLProcessor.scala:248)


[error]


org.apache.daffodil.tdml.processor.DaffodilTDMLDFDLProcessor.parse(DaffodilTDMLDFDLProcessor.scala:253),


took 0.003 sec
[error]     at
org.apache.daffodil.exceptions.Assert$.abort(Assert.scala:137)
[error]     at


org.apache.daffodil.tdml.processor.DaffodilTDMLDFDLProcessor.doParseWithBothApis(DaffodilTDMLDFDLProcessor.scala:290)


[error]     at


org.apache.daffodil.tdml.processor.DaffodilTDMLDFDLProcessor.parse(DaffodilTDMLDFDLProcessor.scala:248)


[error]     at


org.apache.daffodil.tdml.processor.DaffodilTDMLDFDLProcessor.parse(DaffodilTDMLDFDLProcessor.scala:253)


[error]     at


org.apache.daffodil.tdml.ParserTestCase.doParseExpectSuccess(TDMLRunner.scala:1026)


[error]     at


org.apache.daffodil.tdml.ParserTestCase.runParseExpectSuccess(TDMLRunner.scala:1169)


[error]     at


org.apache.daffodil.tdml.ParserTestCase.$anonfun$runProcessor$2(TDMLRunner.scala:920)


[error]     at


org.apache.daffodil.tdml.ParserTestCase.$anonfun$runProcessor$2$adapted(TDMLRunner.scala:917)


[error]     at
scala.util.Either$RightProjection.foreach(Either.scala:652)
[error]     at


org.apache.daffodil.tdml.ParserTestCase.runProcessor(TDMLRunner.scala:917)

[error]     at
org.apache.daffodil.tdml.TestCase.run(TDMLRunner.scala:870)
[error]     at

org.apache.daffodil.tdml.DFDLTestSuite.runOneTest(TDMLRunner.scala:416)
[error]     at
org.apache.daffodil.tdml.Runner.runOneTest(RunnerFactory.scala:191)
[error]     at
org.apache.daffodil.tdml.Runner.runOneTest(RunnerFactory.scala:197)