This is an automated email from the ASF dual-hosted git repository.
slawrence pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-daffodil.git
The following commit(s) were added to refs/heads/master by this push:
new bfe0a9d Evaluate suspensions earlier
bfe0a9d is described below
commit bfe0a9d03efb41224ed5b4a5324eb31a23fb3709
Author: Steve Lawrence <[email protected]>
AuthorDate: Wed Sep 30 08:04:16 2020 -0400
Evaluate suspensions earlier
- The goal here is to evaluate suspensions much earlier during an
unparse rather than waiting to the very end. This should hopefully
allow Daffodil to release buffered data and reduce the memory overhead
required while parsing some files.
- The implementation is inspired by how Java handles garbage collection.
A new SuspensionTracker is add to maintain two lists of suspensions, a
young and an old. The young list is evaluated shortly after the
suspension is created. If a young suspension fails to evaluate, we
bump it to the old suspension list. We evaluate old suspensions much
less frequently. This logic allows us to hopefully reduce buffering
without adding much overhead for evaluating suspensions that are
blocked for a while, all without making the code too complex.
- This SuspensionTracker outputs statistics in debug mode so we can
easily determine how many suspensions were created and how many times
suspensions were evaluated. Ideally the numbers would be the same.
- Add new Assert.invariant macro to provide a custom error message when an
invariant fails
These changes also revealed a few bugs related to unparsing. The
following changes were to resolve those issues:
- Remove the ability to create nested suspensions. It is possible for
the optional separator suspension to require a mandatory text
alignment suspension, which means it's possible to have nested
suspensions, which breaks assumptions of how our data output streams
work. Rather than trying to support this, this uncouples the MTA and
separator unparsers and adds special logic to handle optional
alignment with optional separators. Additionally, add test showing
that MTA is correctly skipped when optional separators are not
unparsed due to a zero length field
- Fix bug where setting the absolute end position state modified the
maybeStartDOS pointer rather than the maybeEndDOS pointer. Because the
maybeDOS pointers are used to determine if the position value
represents an absolute or relative position, this bug made is so
setting the absolute end position changed a relative start position
into absolute. This broke some length calculations since they then
thought they knew were an element starts.
- Fix bug when delivering buffered content. We currently use the format
that causes the direct DOS to be finished to deliver all following
finished DOS's. But it's possible that the bitOrder's of DOS can
change and thus this FormatInfo isn't correct for following DOSs.
Instead, when a DOS is finished, we save the FormatInfo last used when
it was finished. Then when delivering DOSs, we use this saved
FormatInfo. This way, we always deliver buffered DOS using the
FormatInfo that was last used to write.
- Fix issue in logging macro. If you did something like:
log(LogLevel.Error, "%d %f", 1, 3.0)
The log macro wraps the args in a Seq(). With no type on the Seq,
Scala will cast the parameters to similar types, so Seq(1, 3.0)
becomes Seq[Double](1.0, 3.0). These types no longer match the types
in the printf style message, and so you get a TypeCast failure. By
specifying Seq[Any] in the logger macro, it disables this casting and
parameters keep their expected types. Add a test for this issue and
refactor logger testing a bit to be more thread safe and clear.
DAFFODIL-1272
---
.../daffodil/grammar/SequenceGrammarMixin.scala | 13 ++-
.../apache/daffodil/grammar/TermGrammarMixin.scala | 2 +-
.../grammar/primitives/SequenceChild.scala | 6 +-
.../grammar/primitives/SequenceCombinator.scala | 56 ++++++++-
.../daffodil/runtime1/SchemaSetRuntime1Mixin.scala | 1 +
.../io/DirectOrBufferedDataOutputStream.scala | 25 +++-
.../org/apache/daffodil/exceptions/Assert.scala | 11 +-
.../scala/org/apache/daffodil/util/Logger.scala | 12 --
.../org/apache/daffodil/util/TestLogger.scala | 87 +++++++-------
.../apache/daffodil/exceptions/AssertMacros.scala | 10 ++
.../org/apache/daffodil/util/LoggerMacros.scala | 2 +-
.../resources/org/apache/daffodil/xsd/dafext.xsd | 26 +++++
.../processors/unparsers/ElementUnparser.scala | 2 +
.../processors/unparsers/FramingUnparsers.scala | 18 ++-
.../unparsers/SeparatedSequenceUnparsers.scala | 84 ++++++++++----
.../unparsers/SequenceUnparserBases.scala | 5 -
.../unparsers/SuppressableSeparatorUnparser.scala | 71 ++++++++----
.../org/apache/daffodil/infoset/InfosetImpl.scala | 4 +-
.../apache/daffodil/processors/DataProcessor.scala | 2 +-
.../daffodil/processors/SuspensionTracker.scala | 127 +++++++++++++++++++++
.../parsers/SeparatedSequenceParsers.scala | 2 +
.../daffodil/processors/unparsers/UState.scala | 57 ++-------
.../section12/aligned_data/Aligned_Data.tdml | 58 ++++++++++
.../section12/aligned_data/TestAlignedData.scala | 2 +
24 files changed, 515 insertions(+), 168 deletions(-)
diff --git
a/daffodil-core/src/main/scala/org/apache/daffodil/grammar/SequenceGrammarMixin.scala
b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/SequenceGrammarMixin.scala
index 642a44d..5c12781 100644
---
a/daffodil-core/src/main/scala/org/apache/daffodil/grammar/SequenceGrammarMixin.scala
+++
b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/SequenceGrammarMixin.scala
@@ -183,7 +183,18 @@ trait SequenceGrammarMixin
*/
lazy val hasSeparator = !separatorParseEv.isConstantEmptyString
+ /**
+ * Note that the sequence separator does not include the delimMTA grammar
+ * like initiators/terminators. This is because unparsing needs to uncouple
+ * MTA and Separator unparsers to properly support optional separators with
+ * potential alignment. Grammars are expected to handle the delimMTA when
+ * necessary
+ */
+ lazy val sequenceSeparatorMTA = prod("sequenceSeparatorMTA", hasSeparator) {
+ delimMTA
+ }
lazy val sequenceSeparator = prod("separator", hasSeparator) {
- delimMTA ~ SequenceSeparator(this)
+ SequenceSeparator(this)
}
+
}
diff --git
a/daffodil-core/src/main/scala/org/apache/daffodil/grammar/TermGrammarMixin.scala
b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/TermGrammarMixin.scala
index 0e7227b..57710af 100644
---
a/daffodil-core/src/main/scala/org/apache/daffodil/grammar/TermGrammarMixin.scala
+++
b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/TermGrammarMixin.scala
@@ -66,7 +66,7 @@ trait TermGrammarMixin
/**
* Mandatory text alignment for delimiters
*/
- protected final lazy val delimMTA = prod(
+ final lazy val delimMTA = prod(
"delimMTA",
{
hasDelimiters
diff --git
a/daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/SequenceChild.scala
b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/SequenceChild.scala
index 7e3410c..cf60870 100644
---
a/daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/SequenceChild.scala
+++
b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/SequenceChild.scala
@@ -92,6 +92,8 @@ abstract class SequenceChild(protected val sq:
SequenceTermBase, child: Term, gr
protected def separatedHelper: SeparatedHelper
protected def unseparatedHelper: UnseparatedHelper
+ protected lazy val sepMtaGram = sq.delimMTA
+
protected lazy val sepGram = {
sscb match {
case _: PositionalLike =>
@@ -101,8 +103,8 @@ abstract class SequenceChild(protected val sq:
SequenceTermBase, child: Term, gr
sq.sequenceSeparator
}
- protected lazy val sepParser = sepGram.parser
- protected lazy val sepUnparser = sepGram.unparser
+ protected lazy val sepParser = (sepMtaGram ~ sepGram).parser
+ protected lazy val sepUnparser = (sepMtaGram ~ sepGram).unparser
final protected def srd = sq.sequenceRuntimeData
final protected def trd = child.termRuntimeData
diff --git
a/daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/SequenceCombinator.scala
b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/SequenceCombinator.scala
index f120f15..f6f9185 100644
---
a/daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/SequenceCombinator.scala
+++
b/daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/SequenceCombinator.scala
@@ -24,6 +24,8 @@ import org.apache.daffodil.processors.parsers._
import org.apache.daffodil.schema.annotation.props.SeparatorSuppressionPolicy
import org.apache.daffodil.processors.unparsers._
import org.apache.daffodil.util.Misc
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.util.MaybeInt
/**
* Base class for all kinds of sequences.
@@ -45,8 +47,27 @@ abstract class SequenceCombinator(sq: SequenceTermBase,
sequenceChildren: Seq[Se
class OrderedSequence(sq: SequenceTermBase, sequenceChildrenArg:
Seq[SequenceChild])
extends SequenceCombinator(sq, sequenceChildrenArg) {
+ private lazy val sepMtaGram = sq.sequenceSeparatorMTA
+ // Note that we actually only ever use one of these depending on
+ // various factors. If there is an optional separator and a suspension is
+ // used to unparse that separator, then we cannot use the sepMtaUnparser
+ // because it results in nested suspensions, which isn't allowed. In that
+ // case, the suspension ends up handling both the optional separator and
+ // alignment using sepMtaAlignmentMaybe.
+ private lazy val (sepMtaAlignmentMaybe, sepMtaUnparserMaybe) =
+ if (sepMtaGram.isEmpty) {
+ (MaybeInt.Nope, Maybe.Nope)
+ } else {
+ (MaybeInt(sq.knownEncodingAlignmentInBits), Maybe(sepMtaGram.unparser))
+ }
+
private lazy val sepGram = sq.sequenceSeparator
- private lazy val sepParser = sepGram.parser
+
+ private lazy val sepParser = (sepMtaGram ~ sepGram).parser
+ // we cannot include the mtaGram in the sepUnparser. This is because the
+ // sepUnparser is run in a suspension, and the mtaGram can result in
+ // suspension, which means nested suspensions. Instead, the unparser handles
+ // the mta parser differently to avoid this
private lazy val sepUnparser = sepGram.unparser
private lazy val sequenceChildren = sequenceChildrenArg.toVector
@@ -82,7 +103,11 @@ class OrderedSequence(sq: SequenceTermBase,
sequenceChildrenArg: Seq[SequenceChi
sq.hasSeparator match {
case true => new OrderedSeparatedSequenceUnparser(
srd,
- sq.separatorSuppressionPolicy, sq.separatorPosition, sepUnparser,
+ sq.separatorSuppressionPolicy,
+ sq.separatorPosition,
+ sepMtaAlignmentMaybe,
+ sepMtaUnparserMaybe,
+ sepUnparser,
childUnparsers)
case false =>
new OrderedUnseparatedSequenceUnparser(
@@ -98,8 +123,27 @@ class UnorderedSequence(sq: SequenceTermBase,
sequenceChildrenArg: Seq[SequenceC
import SeparatedSequenceChildBehavior._
+ private lazy val sepMtaGram = sq.delimMTA
+ // Note that we actually only ever use one of these depending on
+ // various factors. If there is an optional separator and a suspension is
+ // used to unparse that separator, then we cannot use the sepMtaUnparser
+ // because it results in nested suspensions, which isn't allowed. In that
+ // case, the suspension ends up handling both the optional separator and
+ // alignment using sepMtaAlignmentMaybe.
+ private lazy val (sepMtaAlignmentMaybe, sepMtaUnparserMaybe) =
+ if (sepMtaGram.isEmpty) {
+ (MaybeInt.Nope, Maybe.Nope)
+ } else {
+ (MaybeInt(sq.knownEncodingAlignmentInBits), Maybe(sepMtaGram.unparser))
+ }
+
private lazy val sepGram = sq.sequenceSeparator
- private lazy val sepParser = sepGram.parser
+
+ private lazy val sepParser = (sepMtaGram ~ sepGram).parser
+ // we cannot include the mtaGram in the sepUnparser. This is because the
+ // sepUnparser is run in a suspension, and the mtaGram can result in
+ // suspension, which means nested suspensions. Instead, the unparser handles
+ // the mta parser differently to avoid this
private lazy val sepUnparser = sepGram.unparser
private lazy val sequenceChildren = sequenceChildrenArg.toVector
@@ -173,7 +217,11 @@ class UnorderedSequence(sq: SequenceTermBase,
sequenceChildrenArg: Seq[SequenceC
sq.hasSeparator match {
case true => new OrderedSeparatedSequenceUnparser(
srd,
- SeparatorSuppressionPolicy.AnyEmpty, sq.separatorPosition,
sepUnparser,
+ SeparatorSuppressionPolicy.AnyEmpty,
+ sq.separatorPosition,
+ sepMtaAlignmentMaybe,
+ sepMtaUnparserMaybe,
+ sepUnparser,
childUnparsers)
case false =>
new OrderedUnseparatedSequenceUnparser(
diff --git
a/daffodil-core/src/main/scala/org/apache/daffodil/runtime1/SchemaSetRuntime1Mixin.scala
b/daffodil-core/src/main/scala/org/apache/daffodil/runtime1/SchemaSetRuntime1Mixin.scala
index ea46146..fb87bf8 100644
---
a/daffodil-core/src/main/scala/org/apache/daffodil/runtime1/SchemaSetRuntime1Mixin.scala
+++
b/daffodil-core/src/main/scala/org/apache/daffodil/runtime1/SchemaSetRuntime1Mixin.scala
@@ -99,6 +99,7 @@ trait SchemaSetRuntime1Mixin { self : SchemaSet =>
// diags.foreach { diag => log(LogLevel.Error, diag.toString())
}
} else {
log(LogLevel.Compile, "Parser = %s.", ssrd.parser.toString)
+ log(LogLevel.Compile, "Unparser = %s.", ssrd.unparser.toString)
log(LogLevel.Compile, "Compilation (DataProcesor) completed with no
errors.")
}
dataProc
diff --git
a/daffodil-io/src/main/scala/org/apache/daffodil/io/DirectOrBufferedDataOutputStream.scala
b/daffodil-io/src/main/scala/org/apache/daffodil/io/DirectOrBufferedDataOutputStream.scala
index 94b5260..3c28a0c 100644
---
a/daffodil-io/src/main/scala/org/apache/daffodil/io/DirectOrBufferedDataOutputStream.scala
+++
b/daffodil-io/src/main/scala/org/apache/daffodil/io/DirectOrBufferedDataOutputStream.scala
@@ -411,8 +411,24 @@ class DirectOrBufferedDataOutputStream private[io] (
Assert.invariant(isDirect)
}
+ /**
+ * We need to keep track of what FormatInfo was used as the last write to
+ * this DOS. This way, when this DOS becomes direct and we start
+ * delivering following buffered DOS's, we are using the correct FormatInfo
+ * to check for bitOrder changes and to write data correctly. We set this
+ * value when this DOS is marked as finished with setFinished() and then use
+ * it when delivering buffered content.
+ */
+ private var finishedFormatInfo: Maybe[FormatInfo] = Nope
+
override def setFinished(finfo: FormatInfo): Unit = {
Assert.usage(!isFinished)
+ Assert.usage(finishedFormatInfo.isEmpty)
+
+ // this DOS is finished, save this format info so we use it when delivering
+ // buffered content
+ finishedFormatInfo = One(finfo)
+
// if we are direct, and there's a buffer following this one
//
// we know it isn't finished (because of flush() above)
@@ -433,9 +449,14 @@ class DirectOrBufferedDataOutputStream private[io] (
first.setAbsStartingBitPos0b(dabp)
}
- DirectOrBufferedDataOutputStream.deliverBufferContent(directStream,
first, finfo) // from first, into direct stream's buffers
+ // from first, into direct stream's buffers. Make sure we use the
+ // format info last used for this DOS.
+ DirectOrBufferedDataOutputStream.deliverBufferContent(
+ directStream,
+ first,
+ directStream.finishedFormatInfo.get)
+
// so now the first one is an EMPTY not necessarily a finished
buffered DOS
- //
first.convertToDirect(directStream) // first is now the direct stream
directStream.setDOSState(Uninitialized) // old direct stream is now
dead
directStream = first // long live the new direct stream!
diff --git
a/daffodil-lib/src/main/scala/org/apache/daffodil/exceptions/Assert.scala
b/daffodil-lib/src/main/scala/org/apache/daffodil/exceptions/Assert.scala
index ef27335..b123b50 100644
--- a/daffodil-lib/src/main/scala/org/apache/daffodil/exceptions/Assert.scala
+++ b/daffodil-lib/src/main/scala/org/apache/daffodil/exceptions/Assert.scala
@@ -84,13 +84,22 @@ object Assert extends Assert {
def usage(testAbortsIfFalse: Boolean): Unit = macro AssertMacros.usageMacro1
/**
- * test for something that the program is supposed to be insuring.
+ * test for something that the program is supposed to be ensuring.
*
* This is for more complex invariants than the simple 'impossible' case.
*/
def invariant(testAbortsIfFalse: Boolean): Unit = macro
AssertMacros.invariantMacro1
/**
+ * test for something that the program is supposed to be ensuring, with a
custom error message.
+ *
+ * This is for more complex invariants than the simple 'impossible' case.
+ *
+ * The msg parameter is only evaluated if the test fails
+ */
+ def invariant(testAbortsIfFalse: Boolean, msg: String): Unit = macro
AssertMacros.invariantMacro2
+
+ /**
* Conditional behavior for NYIs
*/
def notYetImplemented(): Nothing = macro AssertMacros.notYetImplementedMacro0
diff --git a/daffodil-lib/src/main/scala/org/apache/daffodil/util/Logger.scala
b/daffodil-lib/src/main/scala/org/apache/daffodil/util/Logger.scala
index 967d9fd..a1c6b6e 100644
--- a/daffodil-lib/src/main/scala/org/apache/daffodil/util/Logger.scala
+++ b/daffodil-lib/src/main/scala/org/apache/daffodil/util/Logger.scala
@@ -96,18 +96,6 @@ abstract class LogWriter {
}
}
-object ForUnitTestLogWriter extends LogWriter {
- var loggedMsg: String = null
- // protected val writer = actor { loop { react { case msg : String =>
- // loggedMsg = msg
- // Console.out.println("Was Logged: " + loggedMsg)
- // Console.out.flush()
- // } } }
- def write(msg: String): Unit = {
- loggedMsg = msg
- }
-}
-
object NullLogWriter extends LogWriter {
//protected val writer = actor { loop { react { case msg : String => } } }
def write(msg: String): Unit = {
diff --git
a/daffodil-lib/src/test/scala/org/apache/daffodil/util/TestLogger.scala
b/daffodil-lib/src/test/scala/org/apache/daffodil/util/TestLogger.scala
index 7c01e1b..fd9ae42 100644
--- a/daffodil-lib/src/test/scala/org/apache/daffodil/util/TestLogger.scala
+++ b/daffodil-lib/src/test/scala/org/apache/daffodil/util/TestLogger.scala
@@ -21,63 +21,72 @@ import org.junit.Assert._
import org.apache.daffodil.exceptions._
import org.junit.Test
-class MyClass extends Logging {
-
- lazy val msg = {
- // System.err.println("computing the message string.")
- "Message %s"
- }
-
- lazy val argString = {
- // System.err.println("computing the argument string.")
- "about nothing at all."
+class ForUnitTestLogWriter extends LogWriter {
+ var loggedMsg: String = null
+ def write(msg: String): Unit = {
+ loggedMsg = msg
}
+}
- lazy val bombArg: String = Assert.abort("bombArg should not be evaluated")
- lazy val bombMsg: String = Assert.abort("bombMsg should not be evaluated")
+class TestLogger {
+ @Test def test1(): Unit = {
- def logSomething(): Unit = {
+ class A extends Logging {
- ForUnitTestLogWriter.loggedMsg = null
+ lazy val msg = "Message %s"
+ lazy val argString = "about nothing at all."
- setLogWriter(ForUnitTestLogWriter)
+ lazy val bombArg: String = Assert.abort("bombArg should not be
evaluated")
+ lazy val bombMsg: String = Assert.abort("bombMsg should not be
evaluated")
- // won't log because below threshhold. Won't even evaluate args.
- log(LogLevel.Debug, msg, bombArg) // Won't show up in log. Won't bomb.
+ def logSomething(): Unit = {
+ // won't log because below threshhold. Won't even evaluate args.
+ log(LogLevel.Debug, msg, bombArg) // Won't show up in log. Won't bomb.
- // alas, no by-name passing of var-args.
- // so instead, we pass by name, a by-name/lazy constructed tuple
- // instead.
+ // alas, no by-name passing of var-args.
+ // so instead, we pass by name, a by-name/lazy constructed tuple
+ // instead.
- // Illustrates that our Glob object, because its parts are all passed by
name,
- // does NOT force evaluation of the pieces that go into it.
- // So it really makes the whole system behave like it was entirely lazy.
+ // Illustrates that our Glob object, because its parts are all passed
by name,
+ // does NOT force evaluation of the pieces that go into it.
+ // So it really makes the whole system behave like it was entirely
lazy.
- // If we're logging below the threshhold of Debug, then this log line
- // doesn't evaluate bombMsg or bombArg. So it is ok if those are expensive
- // to compute.
+ // If we're logging below the threshhold of Debug, then this log line
+ // doesn't evaluate bombMsg or bombArg. So it is ok if those are
expensive
+ // to compute.
- log(LogLevel.Debug, bombMsg, bombArg) // bomb is not evaluated at all.
- log(LogLevel.Error, msg, argString) // Will show up in log.
+ log(LogLevel.Debug, bombMsg, bombArg) // bomb is not evaluated at all.
+ log(LogLevel.Error, msg, argString) // Will show up in log.
+ }
+ }
- setLoggingLevel(LogLevel.Info)
- setLogWriter(ConsoleWriter)
- }
-}
-class TestLogger {
-
- @Test def test1(): Unit = {
- val c = new MyClass
+ val lw = new ForUnitTestLogWriter
+ val c = new A
c.setLoggingLevel(LogLevel.Error)
+ c.setLogWriter(lw)
c.logSomething()
- Console.out.flush()
- val fromLog = ForUnitTestLogWriter.loggedMsg
- // println(fromLog)
+ val fromLog = lw.loggedMsg
val hasExpected = fromLog.contains("Message about nothing at all.")
val doesntHaveUnexpected = !fromLog.contains("number 1")
assertTrue(hasExpected)
assertTrue(doesntHaveUnexpected)
}
+ @Test def test_var_args_different_primitives(): Unit = {
+
+ class A extends Logging {
+ def logSomething(): Unit = {
+ log(LogLevel.Error, "Message: int=%d float=%f", 1, 3.0)
+ }
+ }
+
+ val lw = new ForUnitTestLogWriter
+ val a = new A
+ a.setLoggingLevel(LogLevel.Error)
+ a.setLogWriter(lw)
+ a.logSomething()
+ assertTrue(lw.loggedMsg.contains("Message: int=1 float=3.0"))
+ }
+
}
diff --git
a/daffodil-macro-lib/src/main/scala/org/apache/daffodil/exceptions/AssertMacros.scala
b/daffodil-macro-lib/src/main/scala/org/apache/daffodil/exceptions/AssertMacros.scala
index 91a40c9..ad69bc3 100644
---
a/daffodil-macro-lib/src/main/scala/org/apache/daffodil/exceptions/AssertMacros.scala
+++
b/daffodil-macro-lib/src/main/scala/org/apache/daffodil/exceptions/AssertMacros.scala
@@ -57,6 +57,16 @@ object AssertMacros {
"""
}
+ def invariantMacro2(c: Context)(testAbortsIfFalse: c.Tree, msg: c.Tree):
c.Tree = {
+ import c.universe._
+
+ q"""
+ if (!($testAbortsIfFalse)) {
+ Assert.abort("Invariant broken. " + { $msg })
+ }
+ """
+ }
+
def notYetImplementedMacro0(c: Context)(): c.Tree = {
import c.universe._
diff --git
a/daffodil-macro-lib/src/main/scala/org/apache/daffodil/util/LoggerMacros.scala
b/daffodil-macro-lib/src/main/scala/org/apache/daffodil/util/LoggerMacros.scala
index 53df4bc..e95c5e4 100644
---
a/daffodil-macro-lib/src/main/scala/org/apache/daffodil/util/LoggerMacros.scala
+++
b/daffodil-macro-lib/src/main/scala/org/apache/daffodil/util/LoggerMacros.scala
@@ -59,7 +59,7 @@ object LoggerMacros {
val $level = $lvl
val $l = $level.lvl
if ($ths.getLoggingLevel().lvl >= $l)
- $ths.doLogging($level, $msg, Seq(..$args))
+ $ths.doLogging($level, $msg, Seq[Any](..$args))
}
"""
}
diff --git
a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dafext.xsd
b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dafext.xsd
index 88aae59..bbfd02b 100644
--- a/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dafext.xsd
+++ b/daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dafext.xsd
@@ -400,6 +400,32 @@
</xs:documentation>
</xs:annotation>
</xs:element>
+ <xs:element name="unparseSuspensionWaitOld" type="xs:int"
default="100" minOccurs="0">
+ <xs:annotation>
+ <xs:documentation>
+ While unparsing, some unparse actions require "suspending" which
+ requires buffering unparse output until the suspension can be
+ evaluated. Daffodil periodically attempts to reevaluate these
+ suspensions so that these buffers can be released. We attempt to
+ evaluate young suspensions shortly after creation with the hope
+ that it will succeed and we can release associated buffers. But
if
+ a young suspension fails it is moved to the old suspension list.
+ Old suspensions are evaluated less frequently since they are less
+ likely to succeeded. This minimizes the overhead related to
+ evaluating suspensions that are likely to fail. The
+ unparseSuspensionWaitYoung and unparseSuspensionWaitOld
+ values determine how many elements are unparsed before evaluating
+ young and old suspensions, respectively.
+ </xs:documentation>
+ </xs:annotation>
+ </xs:element>
+ <xs:element name="unparseSuspensionWaitYoung" type="xs:int"
default="5" minOccurs="0">
+ <xs:annotation>
+ <xs:documentation>
+ See unparseSuspensionWaitOld
+ </xs:documentation>
+ </xs:annotation>
+ </xs:element>
</xs:all>
</xs:complexType>
</xs:element>
diff --git
a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/ElementUnparser.scala
b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/ElementUnparser.scala
index 2b92952..bf23800 100644
---
a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/ElementUnparser.scala
+++
b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/ElementUnparser.scala
@@ -481,6 +481,8 @@ sealed trait RegularElementUnparserStartEndStrategy
state.currentInfosetNodeStack.pop
move(state)
+
+ state.asInstanceOf[UStateMain].evalSuspensions(isFinal = false)
}
}
diff --git
a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/FramingUnparsers.scala
b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/FramingUnparsers.scala
index 3c26012..9eb0632 100644
---
a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/FramingUnparsers.scala
+++
b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/FramingUnparsers.scala
@@ -35,12 +35,12 @@ class SkipRegionUnparser(
}
}
-class AlignmentFillUnparserSuspendableOperation(
- alignmentInBits: Int,
- override val rd: TermRuntimeData)
- extends SuspendableOperation {
+trait AlignmentFillUnparserSuspendableMixin { this: SuspendableOperation =>
+
+ def alignmentInBits: Int
+ def rd: TermRuntimeData
- override def test(ustate: UState) = {
+ def test(ustate: UState) = {
val dos = ustate.dataOutputStream
if (dos.maybeAbsBitPos0b.isEmpty) {
log(LogLevel.Debug, "%s %s Unable to align to %s bits because there is
no absolute bit position.", this, ustate, alignmentInBits)
@@ -48,7 +48,7 @@ class AlignmentFillUnparserSuspendableOperation(
dos.maybeAbsBitPos0b.isDefined
}
- override def continuation(state: UState): Unit = {
+ def continuation(state: UState): Unit = {
val dos = state.dataOutputStream
val b4 = dos.relBitPos0b
if (!dos.align(alignmentInBits, state))
@@ -62,6 +62,12 @@ class AlignmentFillUnparserSuspendableOperation(
}
}
+class AlignmentFillUnparserSuspendableOperation(
+ override val alignmentInBits: Int,
+ override val rd: TermRuntimeData)
+ extends SuspendableOperation
+ with AlignmentFillUnparserSuspendableMixin
+
class AlignmentFillUnparser(
alignmentInBits: Int,
override val context: TermRuntimeData)
diff --git
a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/SeparatedSequenceUnparsers.scala
b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/SeparatedSequenceUnparsers.scala
index bd0d393..89ccc73 100644
---
a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/SeparatedSequenceUnparsers.scala
+++
b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/SeparatedSequenceUnparsers.scala
@@ -24,6 +24,8 @@ import org.apache.daffodil.processors.ModelGroupRuntimeData
import scala.collection.mutable.Buffer
import SeparatorSuppressionPolicy._
import SeparatorPosition._
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.util.MaybeInt
trait Separated { self: SequenceChildUnparser =>
@@ -99,9 +101,11 @@ class OrderedSeparatedSequenceUnparser(
rd: SequenceRuntimeData,
ssp: SeparatorSuppressionPolicy,
spos: SeparatorPosition,
+ sepMtaAlignmentMaybe: MaybeInt,
+ sepMtaUnparserMaybe: Maybe[Unparser],
sep: Unparser,
childUnparsersArg: Vector[SequenceChildUnparser])
- extends OrderedSequenceUnparserBase(rd, childUnparsersArg :+ sep) {
+ extends OrderedSequenceUnparserBase(rd, (childUnparsersArg :+ sep) ++
sepMtaUnparserMaybe.toSeq) {
private val childUnparsers =
childUnparsersArg.asInstanceOf[Seq[SequenceChildUnparser with Separated]]
@@ -117,17 +121,18 @@ class OrderedSeparatedSequenceUnparser(
if (trd.isRepresented) {
spos match {
case Prefix => {
- sep.unparse1(state)
+ unparseJustSeparator(state)
unparser.unparse1(state)
}
case Infix => {
- if (state.groupPos > 1)
- sep.unparse1(state)
+ if (state.groupPos > 1) {
+ unparseJustSeparator(state)
+ }
unparser.unparse1(state)
}
case Postfix => {
unparser.unparse1(state)
- sep.unparse1(state)
+ unparseJustSeparator(state)
}
}
} else {
@@ -137,31 +142,56 @@ class OrderedSeparatedSequenceUnparser(
}
/**
- * Unparses the separator only.
+ * Unparses just the separator, as well as any mandatory text alignment if
necessary
*
* Does not deals with infix boundary condition.
*/
private def unparseJustSeparator(state: UState): Unit = {
+ if (sepMtaUnparserMaybe.isDefined) {
+ // we know we are unparsing a separator here, so we must also unparse
+ // mandatory text alignment for that separator. If we didn't staticaly
+ // determine MTA isn't necessary, we must unparse the MTA. This might
+ // lead to a suspension, which is okay in this case because this logic is
+ // not in a suspension, so nested suspensions are avoided.
+ sepMtaUnparserMaybe.get.unparse1(state)
+ }
sep.unparse1(state)
}
/**
- * Unparses the separator only.
+ * Unparses the separator only, which might be optional. The suspension
+ * handles determine if the separator should be unparsed as well as if
+ * alignment is needed, and avoids issues with nested suspensions.
*
* Does not have to deal with infix and first child.
+ *
+ * FIXME: this function is not used anywhere and appears to be dead code.
+ * This is commented out for now so as to not affect code coverage. See
+ * DAFFODIL-2405 and potentially related DAFFODIL-2219 to determine the
+ * future of this code.
*/
- private def unparseJustSeparatorWithTrailingSuppression(
- trd: TermRuntimeData,
- state: UState,
- trailingSuspendedOps:
Buffer[SuppressableSeparatorUnparserSuspendableOperation]): Unit = {
-
- val suspendableOp = new
SuppressableSeparatorUnparserSuspendableOperation(sep, trd)
- // TODO: merge these two objects. We can allocate just one thing here.
- val suppressableSep = SuppressableSeparatorUnparser(sep, trd,
suspendableOp)
-
- suppressableSep.unparse1(state)
- trailingSuspendedOps += suspendableOp
- }
+//private def unparseJustSeparatorWithTrailingSuppression(
+// trd: TermRuntimeData,
+// state: UState,
+// trailingSuspendedOps:
Buffer[SuppressableSeparatorUnparserSuspendableOperation]): Unit = {
+//
+// // We don't know if the unparse will result in zero length or not. We have
+// // to use a suspendable unparser here for the separator which suspends
+// // until it is known whether the unparse of the contents were ZL or not. If
+// // the suspension determines that the field is non-zero length then the
+// // suspenion must also unparser mandatory text alignment for the separator.
+// // This cannot be done with a standard MTA alignment unparser since that is
+// // a suspension and suspensions cannot create suspensions. This this
+// // suspension is also responsible for unparsing alignment if the separator
+// // should be unparsed.
+//
+// val suspendableOp = new
SuppressableSeparatorUnparserSuspendableOperation(sepMtaAlignmentMaybe, sep,
trd)
+// // TODO: merge these two objects. We can allocate just one thing here.
+// val suppressableSep = SuppressableSeparatorUnparser(sep, trd,
suspendableOp)
+//
+// suppressableSep.unparse1(state)
+// trailingSuspendedOps += suspendableOp
+//}
/**
* Unparses an entire sequence, including both scalar and array/optional
children.
@@ -182,10 +212,15 @@ class OrderedSeparatedSequenceUnparser(
trailingSuspendedOps:
Buffer[SuppressableSeparatorUnparserSuspendableOperation],
onlySeparatorFlag: Boolean): Unit = {
val doUnparseChild = !onlySeparatorFlag
- // We don't know if the unparse will result in zero length or not.
- // We have to use a suspendable unparser here for the separator
- // which suspends until it is known whether the unparse of the contents
- // were ZL or not.
+ // We don't know if the unparse will result in zero length or not. We have
+ // to use a suspendable unparser here for the separator which suspends
+ // until it is known whether the unparse of the contents were ZL or not. If
+ // the suspension determines that the field is non-zero length then the
+ // suspension must also unparse mandatory text alignment for the separator.
+ // This cannot be done with a standard MTA alignment unparser since that is
+ // a suspension and suspensions cannot create suspensions. This suspension
+ // is also responsible for unparsing alignment if the separator should be
+ // unparsed.
//
// infix, prefix, postfix matters here, because the separator comes after
// for postfix.
@@ -194,8 +229,7 @@ class OrderedSeparatedSequenceUnparser(
// no separator possible; hence, no suppression
if (doUnparseChild) unparser.unparse1(state)
} else {
-
- val suspendableOp = new
SuppressableSeparatorUnparserSuspendableOperation(sep, trd)
+ val suspendableOp = new
SuppressableSeparatorUnparserSuspendableOperation(sepMtaAlignmentMaybe, sep,
trd)
// TODO: merge these two objects. We can allocate just one thing here.
val suppressableSep = SuppressableSeparatorUnparser(sep, trd,
suspendableOp)
diff --git
a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/SequenceUnparserBases.scala
b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/SequenceUnparserBases.scala
index d510b50..a9c4c4f 100644
---
a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/SequenceUnparserBases.scala
+++
b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/SequenceUnparserBases.scala
@@ -34,9 +34,4 @@ abstract class OrderedSequenceUnparserBase(
// Sequences of nothing (no initiator, no terminator, nothing at all) should
// have been optimized away
Assert.invariant(childUnparsers.length > 0)
-
- // Since some of the grammar terms might have folded away to EmptyGram,
- // the number of unparsers here may be different from the number of
- // children of the sequence group.
- Assert.invariant(srd.groupMembers.length >= childUnparsers.length - 1) //
minus 1 for the separator unparser
}
diff --git
a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/SuppressableSeparatorUnparser.scala
b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/SuppressableSeparatorUnparser.scala
index 4455f8a..beeb24e 100644
---
a/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/SuppressableSeparatorUnparser.scala
+++
b/daffodil-runtime1-unparser/src/main/scala/org/apache/daffodil/processors/unparsers/SuppressableSeparatorUnparser.scala
@@ -23,6 +23,7 @@ import org.apache.daffodil.util.Maybe
import org.apache.daffodil.io.DataOutputStream
import org.apache.daffodil.io.ZeroLengthStatus
import org.apache.daffodil.processors.Processor
+import org.apache.daffodil.util.MaybeInt
/**
* Performance Note: This can be a very special purpose suspension. Unlike the
@@ -32,10 +33,16 @@ import org.apache.daffodil.processors.Processor
* But we need none of the state needed to unparse or evaluate expressions.
*/
final class SuppressableSeparatorUnparserSuspendableOperation(
+ sepMtaAlignmentMaybe: MaybeInt,
sepUnparser: Unparser,
override val rd: TermRuntimeData)
extends SuspendableOperation
- with StreamSplitter {
+ with StreamSplitter
+ with AlignmentFillUnparserSuspendableMixin {
+
+ override val alignmentInBits =
+ if (sepMtaAlignmentMaybe.isDefined) sepMtaAlignmentMaybe.get
+ else 0
private var zlStatus_ : ZeroLengthStatus = ZeroLengthStatus.Unknown
@@ -106,28 +113,42 @@ final class
SuppressableSeparatorUnparserSuspendableOperation(
* finished and length is still zero the test will also return true.
Otherwise they
* are possibly just temporarily of length zero, so we don't know so we
return false
* and the suspension will be retried later.
+ *
+ * Also note that this must take into account alignment. However, we only
+ * care about alignment if we will create a separator, which only occurs when
+ * the zero length status is NonZero. If we determine the zero length status
+ * is Zero, no separator will be unparsed, and so MTA should also not be
+ * unparsed.
*/
override def test(ustate: UState): Boolean = {
- if (zlStatus_ ne ZeroLengthStatus.Unknown)
- true
- else if (maybeDOSAfterSeparatorRegion.isEmpty)
- false
- else {
- Assert.invariant(maybeDOSAfterSeparatorRegion.isDefined)
- if (dosToCheck_.exists { dos =>
- val dosZLStatus = dos.zeroLengthStatus
- dosZLStatus eq ZeroLengthStatus.NonZero
- }) {
- zlStatus_ = ZeroLengthStatus.NonZero
- true
- } else if (dosToCheck_.forall { dos =>
- val dosZLStatus = dos.zeroLengthStatus
- dosZLStatus eq ZeroLengthStatus.Zero
- }) {
- zlStatus_ = ZeroLengthStatus.Zero
+
+ // mutate zlStatus state depending on dos associated with this suspension
+ if ((zlStatus_ ne ZeroLengthStatus.Unknown) ||
maybeDOSAfterSeparatorRegion.isEmpty) {
+ // no-op, we have either already calculated the zls or we don't have a
+ // final DOS yet so can't try to calculate
+ } else if (dosToCheck_.exists { _.zeroLengthStatus eq
ZeroLengthStatus.NonZero }) {
+ zlStatus_ = ZeroLengthStatus.NonZero
+ } else if (dosToCheck_.forall { _.zeroLengthStatus eq
ZeroLengthStatus.Zero }) {
+ zlStatus_ = ZeroLengthStatus.Zero
+ }
+
+ zlStatus_ match {
+ case ZeroLengthStatus.Zero => {
+ // zero length, so there is no separator, so there is nothing to do
true
- } else {
- Assert.invariant(zlStatus_ eq ZeroLengthStatus.Unknown)
+ }
+ case ZeroLengthStatus.NonZero => {
+ // non zero length, so there is a separator. In adition to handling the
+ // separator, this suspension also handles the mandatory text alignment
+ // for that separator to avoid nested suspensions. This suspension test
+ // passes only if we have statically determined that mta alignment is
+ // not needed because we are already aligned, or the alignment test
+ // passes (vai super.test)
+ sepMtaAlignmentMaybe.isEmpty || super.test(ustate)
+ }
+ case ZeroLengthStatus.Unknown => {
+ // we don't have an answer about the if the separator is needed yet,
+ // the test fails until we can figure out an answer
false
}
}
@@ -140,6 +161,9 @@ final class
SuppressableSeparatorUnparserSuspendableOperation(
* If we're positional and potentially trailing, then this will only be Zero
length
* if we're considering unparsing a trailing separator for an empty, with
nothing following.
* So if ZL, no separator, otherwise we unparse the separator.
+ *
+ * If we are unparsing a separator, we must also unparse associated MTA
+ * alignment if we didn't statically determine that it wasn't needed
*/
override def continuation(state: UState): Unit = {
import ZeroLengthStatus._
@@ -152,6 +176,13 @@ final class
SuppressableSeparatorUnparserSuspendableOperation(
}
case NonZero => {
// non-zero case. So we need the separator.
+
+ // first unparse alignment bits if alignment is necessary
+ if (sepMtaAlignmentMaybe.isDefined) {
+ super.continuation(state)
+ }
+
+ // then unparse the separator
sepUnparser.unparse1(savedUstate)
}
case Unknown =>
diff --git
a/daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/InfosetImpl.scala
b/daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/InfosetImpl.scala
index 51b31cb..4f5a3d1 100644
---
a/daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/InfosetImpl.scala
+++
b/daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/InfosetImpl.scala
@@ -443,7 +443,7 @@ sealed abstract class LengthState(ie: DIElement)
def isEndUndef = {
val r = maybeEndPos0bInBits.isEmpty
- if (r) Assert.invariant(maybeStartDataOutputStream.isEmpty)
+ if (r) Assert.invariant(maybeEndDataOutputStream.isEmpty)
r
}
@@ -583,7 +583,7 @@ sealed abstract class LengthState(ie: DIElement)
def setAbsEndPos0bInBits(absPosInBits0b: ULong): Unit = {
maybeEndPos0bInBits = MaybeULong(absPosInBits0b.longValue)
- maybeStartDataOutputStream = Nope
+ maybeEndDataOutputStream = Nope
}
def setRelEndPos0bInBits(relPosInBits0b: ULong, dos: DataOutputStream): Unit
= {
diff --git
a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/DataProcessor.scala
b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/DataProcessor.scala
index 80fe696..f8939e3 100644
---
a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/DataProcessor.scala
+++
b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/DataProcessor.scala
@@ -575,7 +575,7 @@ class DataProcessor private (
unparserState.dataProc.get.init(ssrd.unparser)
out.setPriorBitOrder(ssrd.elementRuntimeData.defaultBitOrder)
doUnparse(unparserState)
- unparserState.evalSuspensions(unparserState) // handles outputValueCalc
that were suspended due to forward references.
+ unparserState.evalSuspensions(isFinal = true)
unparserState.unparseResult
} catch {
case ue: UnparseError => {
diff --git
a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/SuspensionTracker.scala
b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/SuspensionTracker.scala
new file mode 100644
index 0000000..7808d29
--- /dev/null
+++
b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/SuspensionTracker.scala
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.processors
+
+import scala.collection.mutable.Queue
+
+import org.apache.daffodil.dsom.RuntimeSchemaDefinitionError
+import org.apache.daffodil.exceptions.Assert
+import org.apache.daffodil.util.LogLevel
+import org.apache.daffodil.util.Logging
+
+class SuspensionTracker(suspensionWaitYoung: Int, suspensionWaitOld: Int)
+ extends Logging {
+
+ private val suspensionsYoung = new Queue[Suspension]
+ private val suspensionsOld = new Queue[Suspension]
+
+ private var count: Int = 0
+
+ private var suspensionStatTracked: Int = 0
+ private var suspensionStatRuns: Int = 0
+
+ def trackSuspension(s: Suspension): Unit = {
+ suspensionsYoung.enqueue(s)
+ suspensionStatTracked += 1
+ }
+
+ /**
+ * Attempts to evaluate suspensions. Old suspensions are evaluated less
+ * frequently than young suspensions. Any young suspensions that fail to
+ * evaluate are moved to the old suspensions list. If we evaluate old
+ * suspensions, we attempt to evaluate them first, with the hope that their
+ * resolution might make the young suspensions more likely to evaluate.
+ */
+ def evalSuspensions(): Unit = {
+ if (count % suspensionWaitOld == 0) {
+ evalSuspensionQueue(suspensionsOld)
+ }
+ if (count % suspensionWaitYoung == 0) {
+ evalSuspensionQueue(suspensionsYoung)
+ while (suspensionsYoung.nonEmpty) {
+ suspensionsOld.enqueue(suspensionsYoung.dequeue)
+ }
+ }
+
+ if (count == suspensionWaitOld) {
+ count = 0
+ } else {
+ count += 1
+ }
+ }
+
+ /**
+ * Evaluates all suspensions until either they are all evaluated or a
+ * deadlock is detected. This moves all young suspensions to the old queue,
+ * and evaluates all old suspensions. If the old queue is non-empty, that
+ * means some suspensions are blocked, likely due to a circular deadlock, and
+ * we output diagnostics.
+ */
+ def requireFinal(): Unit = {
+ while (suspensionsYoung.nonEmpty) {
+ suspensionsOld.enqueue(suspensionsYoung.dequeue)
+ }
+
+ evalSuspensionQueue(suspensionsOld)
+
+ Assert.invariant(
+ suspensionsOld.length != 1,
+ "Single suspended expression making no forward progress. " +
suspensionsOld(0))
+
+ if (suspensionsOld.nonEmpty) {
+ throw new SuspensionDeadlockException(suspensionsOld.seq)
+ }
+
+ log(
+ LogLevel.Debug,
+ "Suspension runs/tracked: %d/%d (%.2f%%)",
+ suspensionStatRuns,
+ suspensionStatTracked,
+ (suspensionStatRuns.toFloat / suspensionStatTracked) * 100)
+ }
+
+ /**
+ * Attempt to evaluate suspensions on the provie queue. Keep repeating the
+ * evaluates as long as some progress is being made. Suspensions that
+ * evaluate sucessfully are removed from the queue. Once suspensions make no
+ * further progress and are all blocked, we return. Blocked suspensions put
+ * back on the same queue.
+ */
+ private def evalSuspensionQueue(queue: Queue[Suspension]): Unit = {
+ var countOfNotMakingProgress = 0
+ while (!queue.isEmpty && countOfNotMakingProgress < queue.length) {
+ val s = queue.dequeue
+ suspensionStatRuns += 1
+ s.runSuspension()
+ if (!s.isDone) queue.enqueue(s)
+ if (s.isDone || s.isMakingProgress) {
+ countOfNotMakingProgress = 0
+ } else {
+ countOfNotMakingProgress += 1
+ }
+ }
+ }
+
+}
+
+class SuspensionDeadlockException(suspExprs: Seq[Suspension])
+ extends RuntimeSchemaDefinitionError(
+ suspExprs(0).rd.schemaFileLocation,
+ suspExprs(0).savedUstate,
+ "Expressions/Unparsers are circularly deadlocked (mutually defined):\n%s",
+ suspExprs.groupBy { _.rd }.mapValues { _(0) }.values.mkString(" - ", "\n -
", ""))
diff --git
a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/parsers/SeparatedSequenceParsers.scala
b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/parsers/SeparatedSequenceParsers.scala
index 12ca7ec..115741a 100644
---
a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/parsers/SeparatedSequenceParsers.scala
+++
b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/parsers/SeparatedSequenceParsers.scala
@@ -25,6 +25,8 @@ trait Separated { self: SequenceChildParser =>
def spos: SeparatorPosition
def trd: TermRuntimeData
def parseResultHelper: SeparatedSequenceChildParseResultHelper
+
+ override def childProcessors: Vector[Processor] = Vector(self.childParser)
:+ sep
import SeparatorPosition._
diff --git
a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/unparsers/UState.scala
b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/unparsers/UState.scala
index 04baed2..30091ae 100644
---
a/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/unparsers/UState.scala
+++
b/daffodil-runtime1/src/main/scala/org/apache/daffodil/processors/unparsers/UState.scala
@@ -22,14 +22,12 @@ import java.nio.CharBuffer
import java.nio.LongBuffer
import scala.Left
-import scala.collection.mutable
import org.apache.daffodil.api.DFDL
import org.apache.daffodil.api.DaffodilTunables
import org.apache.daffodil.api.DataLocation
import org.apache.daffodil.api.Diagnostic
import org.apache.daffodil.dpath.UnparserBlocking
-import org.apache.daffodil.dsom.RuntimeSchemaDefinitionError
import org.apache.daffodil.equality.EqualitySuppressUnusedImportWarning
import org.apache.daffodil.exceptions.Assert
import org.apache.daffodil.exceptions.SavesErrorsAndWarnings
@@ -50,6 +48,7 @@ import org.apache.daffodil.processors.Failure
import org.apache.daffodil.processors.NonTermRuntimeData
import org.apache.daffodil.processors.ParseOrUnparseState
import org.apache.daffodil.processors.Suspension
+import org.apache.daffodil.processors.SuspensionTracker
import org.apache.daffodil.processors.TermRuntimeData
import org.apache.daffodil.processors.UnparseResult
import org.apache.daffodil.processors.VariableBox
@@ -444,7 +443,6 @@ final class UStateMain private (
diagnosticsArg: List[Diagnostic],
dataProcArg: DataProcessor,
dos: DirectOrBufferedDataOutputStream,
- initialSuspendedExpressions: mutable.Queue[Suspension],
tunable: DaffodilTunables)
extends UState(dos, vbox, diagnosticsArg, One(dataProcArg), tunable) {
@@ -456,10 +454,9 @@ final class UStateMain private (
diagnosticsArg: List[Diagnostic],
dataProcArg: DataProcessor,
dataOutputStream: DirectOrBufferedDataOutputStream,
- initialSuspendedExpressions: mutable.Queue[Suspension],
tunable: DaffodilTunables) =
this(inputter, new VariableBox(vmap), diagnosticsArg, dataProcArg,
- dataOutputStream, initialSuspendedExpressions, tunable)
+ dataOutputStream, tunable)
private var _prior: UStateForSuspension = null
override def prior = _prior
@@ -609,43 +606,18 @@ final class UStateMain private (
* All the other clones used for outputValueCalc, those never
* need to add any.
*/
- private val suspensions = initialSuspendedExpressions
+ private val suspensionTracker =
+ new SuspensionTracker(
+ tunable.unparseSuspensionWaitYoung,
+ tunable.unparseSuspensionWaitOld)
def addSuspension(se: Suspension): Unit = {
- suspensions.enqueue(se)
+ suspensionTracker.trackSuspension(se)
}
- def evalSuspensions(ustate: UState): Unit = {
- var countOfNotMakingProgress = 0
- while (!suspensions.isEmpty &&
- countOfNotMakingProgress < suspensions.length) {
- val se = suspensions.dequeue
- se.runSuspension()
- if (!se.isDone) suspensions.enqueue(se)
- if (se.isDone || se.isMakingProgress)
- countOfNotMakingProgress = 0
- else
- countOfNotMakingProgress += 1
- }
- // after the loop, did we terminate
- // with some expressions still unevaluated?
- if (suspensions.length > 1) {
- // unable to evaluate all the expressions
- suspensions.map { sus =>
- sus.runSuspension() // good place for a breakpoint so we can debug why
things are locked up.
- sus.explain()
- }
- System.err.println("Dump of UStates")
- var us = ustate
- while (us ne null) {
- System.err.println(us)
- us = us.prior
- }
-
- throw new SuspensionDeadlockException(suspensions.seq)
- } else if (suspensions.length == 1) {
- Assert.invariantFailed("Single suspended expression making no forward
progress. " + suspensions(0))
- }
+ def evalSuspensions(isFinal: Boolean): Unit = {
+ suspensionTracker.evalSuspensions()
+ if (isFinal) suspensionTracker.requireFinal()
}
final override def pushTRD(trd: TermRuntimeData) =
@@ -670,13 +642,6 @@ final class UStateMain private (
}
}
-class SuspensionDeadlockException(suspExprs: Seq[Suspension])
- extends RuntimeSchemaDefinitionError(
- suspExprs(0).rd.schemaFileLocation,
- suspExprs(0).savedUstate,
- "Expressions/Unparsers are circularly deadlocked (mutually defined):\n%s",
- suspExprs.groupBy { _.rd }.mapValues { _(0) }.values.mkString(" - ", "\n -
", ""))
-
object UState {
def createInitialUState(
@@ -693,7 +658,7 @@ object UState {
val diagnostics = Nil
val newState = new UStateMain(inputter, variables, diagnostics,
dataProc.asInstanceOf[DataProcessor], out,
- new mutable.Queue[Suspension], dataProc.getTunables()) // null means no
prior UState
+ dataProc.getTunables()) // null means no prior UState
newState
}
}
diff --git
a/daffodil-test/src/test/resources/org/apache/daffodil/section12/aligned_data/Aligned_Data.tdml
b/daffodil-test/src/test/resources/org/apache/daffodil/section12/aligned_data/Aligned_Data.tdml
index 610dbfd..b3ef9e0 100644
---
a/daffodil-test/src/test/resources/org/apache/daffodil/section12/aligned_data/Aligned_Data.tdml
+++
b/daffodil-test/src/test/resources/org/apache/daffodil/section12/aligned_data/Aligned_Data.tdml
@@ -3681,4 +3681,62 @@
</tdml:errors>
</tdml:unparserTestCase>
+
+ <tdml:defineSchema name="separatorMTA">
+ <xs:include
schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>
+
+ <dfdl:format ref="ex:GeneralFormat"
+ lengthUnits="bits"
+ alignmentUnits="bits"
+ fillByte="%#rFF;"/>
+
+ <xs:element name="e1">
+ <xs:complexType>
+ <xs:sequence
+ dfdl:separator=","
+ dfdl:separatorPosition="prefix"
+ dfdl:separatorSuppressionPolicy="anyEmpty"
+ dfdl:encoding="US-ASCII" dfdl:alignment="8">
+ <xs:element name="a" type="xs:string" maxOccurs="unbounded"
+ dfdl:lengthKind="pattern" dfdl:lengthPattern="."
+ dfdl:encoding="X-DFDL-BASE4-MSBF" />
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+
+ </tdml:defineSchema>
+
+ <!--
+ Test Name: separatorMTA_01
+ Schema: separatorMTA
+ Root: root
+ Purpose: This test demonstrates that with the correct separator
+ suppression, separators are only unparsed when elements unparse to
+ non-zero length. Infoset elements that unparse to zero length
+ essentially disappear. This also tests that when we determine a field
+ is zero length and do not unparse the seprator, that we also do not
+ unparse mandatory text alignment associated with that separator
+ -->
+
+ <tdml:unparserTestCase name="separatorMTA_01"
+ model="separatorMTA"
+ description="Section 12.1 - Aligned Data" roundTrip="none">
+ <tdml:infoset>
+ <tdml:dfdlInfoset>
+ <ex:e1>
+ <ex:a>0</ex:a>
+ <ex:a></ex:a>
+ <ex:a>1</ex:a>
+ <ex:a></ex:a>
+ </ex:e1>
+ </tdml:dfdlInfoset>
+ </tdml:infoset>
+ <tdml:document>
+ <tdml:documentPart type="bits">00101100</tdml:documentPart> <!--
us-ascii comma separator -->
+ <tdml:documentPart type="bits">00 111111</tdml:documentPart> <!-- base-4
"0" string + mta fill bits-->
+ <tdml:documentPart type="bits">00101100</tdml:documentPart> <!--
us-ascii comma separator -->
+ <tdml:documentPart type="bits">01</tdml:documentPart> <!-- base-4 "1"
string (no mta fill bits)-->
+ </tdml:document>
+ </tdml:unparserTestCase>
+
</tdml:testSuite>
diff --git
a/daffodil-test/src/test/scala/org/apache/daffodil/section12/aligned_data/TestAlignedData.scala
b/daffodil-test/src/test/scala/org/apache/daffodil/section12/aligned_data/TestAlignedData.scala
index e664ca6..058ff25 100644
---
a/daffodil-test/src/test/scala/org/apache/daffodil/section12/aligned_data/TestAlignedData.scala
+++
b/daffodil-test/src/test/scala/org/apache/daffodil/section12/aligned_data/TestAlignedData.scala
@@ -191,4 +191,6 @@ class TestAlignedData {
@Test def test_fillByte_04() = { runner1.runOneTest("fillByte_04") }
@Test def test_fillByte_05() = { runner1.runOneTest("fillByte_05") }
@Test def test_fillByte_06() = { runner1.runOneTest("fillByte_06") }
+
+ @Test def test_separatorMTA_01() = { runner1.runOneTest("separatorMTA_01") }
}
