[GitHub] mbeckerle commented on a change in pull request #163: Fix performance issues with recent hexBinary changes

[GitBox]

mbeckerle commented on a change in pull request #163: Fix performance issues 
with recent hexBinary changes
URL: https://github.com/apache/incubator-daffodil/pull/163#discussion_r247679360
 
 

 ##
 File path: 
daffodil-io/src/main/scala/org/apache/daffodil/io/InputSourceDataInputStream.scala
 ##
 @@ -156,170 +156,125 @@ final class InputSourceDataInputStream private (val 
inputSource: InputSource)
 
   /**
* Accepts a preallocated array and a bitLength. Reads the specified number
-   * of bits and stores them in the array in big endian byte order and most
-   * significant bit first bit order. The most significant byte is stored in
-   * the zero'th index in the array. This means that if the array is larger
-   * than the number of bytes needed for the specified number of bits, the
-   * trailing bytes will be untouched and should be ignored by the caller.
+   * of bits and stores them in the array based on byteOrder and bitOrder,
+   * resulting in an array consistent with hexBinary data. If there are
+   * fragment bytes the last bits are placed appropriately based on bitOrder.
+   * For example, assuming a bitLength of 19, the resulting array will be
+   * filled like so,
+   *
+   *   BE MSBF: 01234567 01234567 012x
+   *   BE LSBF: 76543210 76543210 x210
+   *
+   * Little endian is the exact same, but with the bytes reversed, like so:
+   *
+   *   LE MSBF: 012x 01234567 01234567
+   *   LE LSBF: x210 76543210 76543210
+   *
+   * In both cases, the x's are always zeros. It is the responsiblity of the
+   * caller to be aware of the length of the data and to know which bits are
+   * data vs padding based on bitOrder/byteOrder
*/
   private def fillByteArray(array: Array[Byte], bitLengthFrom1: Int, finfo: 
FormatInfo): Unit = {
-if (isAligned(8) && bitLengthFrom1 % 8 == 0) {
-  fillByteArrayAlignedNoFragment(array, bitLengthFrom1, finfo)
-} else {
-  fillByteArrayUnalignedOrFragment(array, bitLengthFrom1, finfo)
-}
-  }
-
-  private def fillByteArrayAlignedNoFragment(array: Array[Byte], 
bitLengthFrom1: Int, finfo: FormatInfo): Unit = {
-// threadCheck()
-Assert.usage(isAligned(8))
-Assert.usage(bitLengthFrom1 % 8 == 0)
-
-val bytesToFill = bitLengthFrom1 / 8
-Assert.invariant(array.size >= bytesToFill)
+val isUnaligned = !isAligned(8)
+val fragmentBits = bitLengthFrom1 % 8
+val bytesToFill = (bitLengthFrom1 + 7) / 8
 
-if (bytesToFill == 1 || // 1 byte is super common case. We don't want to 
retrieve byteOrder nor bitOrder in this case
-  (finfo.byteOrder == ByteOrder.BigEndian && finfo.bitOrder == 
BitOrder.MostSignificantBitFirst)) {
-  // bits & bytes are already in order, read them straight into the array
-  inputSource.get(array, 0, bytesToFill)
-} else {
-  // we are either LittleEndian & MSBF or BigEndian & LSBF. In either case,
-  // we just need to flip the bytes to make it BigEndian MSBF. The bits are
-  // in the correct order.
-  var i = bytesToFill - 1
-  while (i >= 0) {
-array(i) = inputSource.get().toByte
-i -= 1
+// We may only need N bytes to fill in the array, but if we are unaligned
+// then that means we could potentially need to get an extra byte. For
+// example, say bitLength is 8, so we only need 1 byte total and the array
+// size is 1. If we are unaligned and our bit offset is 1, we need to read
+// two bytes total, extracting 7 bits from the first byte and 1 bit from
+// the second byte. To be efficient, let's fill the array with data we know
+// we'll need. Later, if we determine that we are unaligned and an extra
+// byte is needed, we will read one more byte into an overflow variable and
+// eventually combine that into the array.
+inputSource.get(array, 0, bytesToFill)
+
+if (isUnaligned) { 
+  // If we are not aligned, then we need to shift all the bits we just got
+  // to the left, based on the bitOffset and the bitOrder. Do that shift
+  // here.
+
+  val isMSBF = finfo.bitOrder == BitOrder.MostSignificantBitFirst
+  val bitOffset0b = (bitPos0b % 8).toInt
+
+  // Determine if we need an overflow byte and read it if so
+  val bytesToRead = (bitLengthFrom1 + bitOffset0b + 7) / 8
+  val arrayOverflow = if (bytesToRead > bytesToFill) 
Bits.asUnsignedByte(inputSource.get()) else 0
+
+  // Determine the masks and shifts needed to create a new byte based on
+  // the bitOrder
+  val curBitMask =
+if (isMSBF)
+  Bits.maskR(8 - bitOffset0b)
+else
+  Bits.maskL(8 - bitOffset0b)
+
+  val nxtBitMask = ~curBitMask & 0xFF
+  val curShift = bitOffset0b
+  val nxtShift = 8 - bitOffset0b
+
+  @inline
 
 Review comment:
   If you @inline a def inside of code like this, do we know if this eliminates 
the closure that would otherwise be allocated?
   
   If so then this is a useful trick to know. 

[GitHub] mbeckerle commented on a change in pull request #163: Fix performance issues with recent hexBinary changes

[GitBox]

mbeckerle commented on a change in pull request #163: Fix performance issues 
with recent hexBinary changes
URL: https://github.com/apache/incubator-daffodil/pull/163#discussion_r247676525
 
 

 ##
 File path: 
daffodil-core/src/main/scala/org/apache/daffodil/dsom/RuntimePropertyMixins.scala
 ##
 @@ -524,34 +536,44 @@ trait ElementRuntimeValuedPropertiesMixin
   ReferencedElementInfos.None
 
   private def localElementPropertyReferencedElements(f: F) = {
 
 Review comment:
   I'm going to submit a ticket about internal code doc improvement needed for 
this f: F parameter. 
   
   I have a hard time understanding this code, but it is not due to your 
changes. It's just lack of good identifiers, and lack of doc for some traits, 
etc. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mbeckerle commented on a change in pull request #163: Fix performance issues with recent hexBinary changes

[GitBox]

mbeckerle commented on a change in pull request #163: Fix performance issues 
with recent hexBinary changes
URL: https://github.com/apache/incubator-daffodil/pull/163#discussion_r247678978
 
 

 ##
 File path: 
daffodil-io/src/main/scala/org/apache/daffodil/io/InputSourceDataInputStream.scala
 ##
 @@ -156,170 +156,125 @@ final class InputSourceDataInputStream private (val 
inputSource: InputSource)
 
   /**
* Accepts a preallocated array and a bitLength. Reads the specified number
-   * of bits and stores them in the array in big endian byte order and most
-   * significant bit first bit order. The most significant byte is stored in
-   * the zero'th index in the array. This means that if the array is larger
-   * than the number of bytes needed for the specified number of bits, the
-   * trailing bytes will be untouched and should be ignored by the caller.
+   * of bits and stores them in the array based on byteOrder and bitOrder,
+   * resulting in an array consistent with hexBinary data. If there are
+   * fragment bytes the last bits are placed appropriately based on bitOrder.
+   * For example, assuming a bitLength of 19, the resulting array will be
+   * filled like so,
+   *
+   *   BE MSBF: 01234567 01234567 012x
+   *   BE LSBF: 76543210 76543210 x210
+   *
+   * Little endian is the exact same, but with the bytes reversed, like so:
+   *
+   *   LE MSBF: 012x 01234567 01234567
 
 Review comment:
   Completely confused by this. So you are numbering bytes left-to-right here?
   
   This is clearer if you put column labels saying Byte 0, Byte 1, Byte 2, Byte 
3 on top of the byte columns.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] mbeckerle commented on a change in pull request #163: Fix performance issues with recent hexBinary changes

[GitBox]

mbeckerle commented on a change in pull request #163: Fix performance issues 
with recent hexBinary changes
URL: https://github.com/apache/incubator-daffodil/pull/163#discussion_r247682262
 
 

 ##
 File path: 
daffodil-io/src/main/scala/org/apache/daffodil/io/InputSourceDataInputStream.scala
 ##
 @@ -156,170 +156,125 @@ final class InputSourceDataInputStream private (val 
inputSource: InputSource)
 
   /**
* Accepts a preallocated array and a bitLength. Reads the specified number
-   * of bits and stores them in the array in big endian byte order and most
-   * significant bit first bit order. The most significant byte is stored in
-   * the zero'th index in the array. This means that if the array is larger
-   * than the number of bytes needed for the specified number of bits, the
-   * trailing bytes will be untouched and should be ignored by the caller.
+   * of bits and stores them in the array based on byteOrder and bitOrder,
 
 Review comment:
   "based on byteOrder and bitOrder, resulting in an array consistent with 
hexBinary data"
   
   I thought hexBinary data ignored byteOrder now.
   
   So this comment is confusing to me. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (DAFFODIL-2051) Many tests are OS-dependent and fail on Windows platform

[Mark C. Otto (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark C. Otto reassigned DAFFODIL-2051:
--

Assignee: Mark C. Otto

Assigned to myself for remediation.

> Many tests are OS-dependent and fail on Windows platform
> 
>
> Key: DAFFODIL-2051
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2051
> Project: Daffodil
>  Issue Type: Test
>  Components: General
>Affects Versions: 2.2.0
> Environment: Dell Lattitude 7450 Laptop, Windows 8.1, Java SDK 
> 11.0.1, SBT 1.2.7, Daffodil 2.2.1
>Reporter: Mark C. Otto
>Assignee: Mark C. Otto
>Priority: Major
>  Labels: test, windows
> Attachments: erroroutput.txt
>
>   Original Estimate: 840h
>  Remaining Estimate: 840h
>
> Many tests are not written in an OS-independent manner. Most of these fail 
> due to OS-dependent line termination differences (LF vs. CR/LF). While some 
> of this was covered in DAFFODIP-1360, that bug was insufficient in scope and 
> detail to cover the issues listed here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (DAFFODIL-2052) code quality: very poor identifiers - private def localElementPropertyReferencedElements(f: F)

[Michael Beckerle (JIRA)]

Michael Beckerle created DAFFODIL-2052:
--

 Summary: code quality: very poor identifiers - private def 
localElementPropertyReferencedElements(f: F)
 Key: DAFFODIL-2052
 URL: https://issues.apache.org/jira/browse/DAFFODIL-2052
 Project: Daffodil
  Issue Type: Improvement
  Components: Clean Ups
Affects Versions: 2.2.0
Reporter: Michael Beckerle
 Fix For: 2.3.0



This method is using a type parameter.

The name is "F". 

This is ineffective at explaining what this is for. Choose better name, and 
enhance this code and surrounding class/trait with scaladoc of this type 
parameter so that we can understand what it does. 

private def localElementPropertyReferencedElements(f: F)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Permission request - DAFFODIL-2051

[Steve Lawrence]

Added.

Thanks,
- Steve

On 1/14/19 4:30 PM, Otto, Mark wrote:
> All:
> 
> I am a new DAFFODIL developer working with Stephen Lawrence and Mike Beckerle 
> at Tresys. I have opened a new bug for a large number of tests that do not 
> work properly on the Windows platform. I am hereby requesting permissions 
> sufficient to modify and assign DAFFODIL-2051 to myself for remediation of 
> these problems.
> 
> Thanks.
> 
> Mark Otto
> Managing Principal, Services
> Tresys Technology
> 8840 Stanford Boulevard Suite 2100
> Columbia, MD 21045
> P: 410.290.1411 x161
> M: +1 408-857-1492
> F: 410.953.0494
> mo...@tresys.com | tresys.com
> 
> 
> 

Permission request - DAFFODIL-2051

[Otto, Mark]

All:

I am a new DAFFODIL developer working with Stephen Lawrence and Mike Beckerle 
at Tresys. I have opened a new bug for a large number of tests that do not work 
properly on the Windows platform. I am hereby requesting permissions sufficient 
to modify and assign DAFFODIL-2051 to myself for remediation of these problems.

Thanks.

Mark Otto
Managing Principal, Services
Tresys Technology
8840 Stanford Boulevard Suite 2100
Columbia, MD 21045
P: 410.290.1411 x161
M: +1 408-857-1492
F: 410.953.0494
mo...@tresys.com | tresys.com

[jira] [Created] (DAFFODIL-2051) Many tests are OS-dependent and fail on Windows platform

[Mark C. Otto (JIRA)]

Mark C. Otto created DAFFODIL-2051:
--

 Summary: Many tests are OS-dependent and fail on Windows platform
 Key: DAFFODIL-2051
 URL: https://issues.apache.org/jira/browse/DAFFODIL-2051
 Project: Daffodil
  Issue Type: Test
  Components: General
Affects Versions: 2.2.0
 Environment: Dell Lattitude 7450 Laptop, Windows 8.1, Java SDK 11.0.1, 
SBT 1.2.7, Daffodil 2.2.1
Reporter: Mark C. Otto
 Attachments: erroroutput.txt

Many tests are not written in an OS-independent manner. Most of these fail due 
to OS-dependent line termination differences (LF vs. CR/LF). While some of this 
was covered in DAFFODIP-1360, that bug was insufficient in scope and detail to 
cover the issues listed here.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (DAFFODIL-2046) Poor handling of missing source attribute on element

[Josh Adams (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Adams resolved DAFFODIL-2046.
--
   Resolution: Fixed
 Assignee: Dave Thompson  (was: Josh Adams)
Fix Version/s: 2.3.0

This issue has been fixed in commit 1ca7e37576b6a8a9c1fd19c2aa5e2dc1db2f8087

> Poor handling of missing source attribute on  element
> -
>
> Key: DAFFODIL-2046
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2046
> Project: Daffodil
>  Issue Type: Bug
>  Components: Diagnostics
>Affects Versions: 2.1.0
>Reporter: Brandon Sloane
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.3.0
>
> Attachments: test.dfdl.xsd
>
>
> When an appinfo element is present inside an empty sequence inside of a 
> choice, and said appinfo element is missing its source attribute, Daffodil 
> crashes instead of producing a useful error message.
>  
> A complete schema is attached, but the relevent section is:
>  
> {quote}
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
>  
> Actual Output:
> !!
> !! An unexpected exception occurred. This is a bug! !!
> !!
> Please report this bug and help us fix it:
> https://daffodil.apache.org/community/#issue-tracker
> Please include the following exception, the command you
>  ran, and any input, schema, or tdml files used that led
>  to this bug.
> org.apache.daffodil.oolag.CircularDefinition: OOLAG Cycle (of 2) through 
> groupMembers@253380088, groupMembers@253380088
>  at 
> org.apache.daffodil.oolag.OOLAG$OOLAGValueBase.oolagBefore(OOLAG.scala:492)
>  at org.apache.daffodil.oolag.OOLAG$OOLAGValue.liftedTree1$1(OOLAG.scala:584)
>  at 
> org.apache.daffodil.oolag.OOLAG$OOLAGValue.value$lzycompute(OOLAG.scala:583)
>  at org.apache.daffodil.oolag.OOLAG$OOLAGValue.value(OOLAG.scala:581)
>  at 
> org.apache.daffodil.dsom.GroupDefLike$class.groupMembers(GroupDef.scala:70)
>  at 
> org.apache.daffodil.dsom.Choice.groupMembers$lzycompute(ChoiceGroup.scala:267)
>  at org.apache.daffodil.dsom.Choice.groupMembers(ChoiceGroup.scala:267)
>  at 
> org.apache.daffodil.dsom.ModelGroup.sequenceChildren$lzycompute(ModelGroup.scala:201)
>  at org.apache.daffodil.dsom.ModelGroup.sequenceChildren(ModelGroup.scala:201)
>  at 
> org.apache.daffodil.dsom.ModelGroup$$anonfun$sequencePeers$1.apply(ModelGroup.scala:161)
>  at 
> org.apache.daffodil.dsom.ModelGroup$$anonfun$sequencePeers$1.apply(ModelGroup.scala:161)
>  at 
> org.apache.daffodil.dsom.ModelGroup.sequencePeers$lzycompute(ModelGroup.scala:161)
>  at org.apache.daffodil.dsom.ModelGroup.sequencePeers(ModelGroup.scala:161)
>  at 
> org.apache.daffodil.dsom.SequenceTermBase.myPeers$lzycompute(SequenceGroup.scala:54)
>  at org.apache.daffodil.dsom.SequenceTermBase.myPeers(SequenceGroup.scala:54)
>  at 
> org.apache.daffodil.dsom.ModelGroup$$anonfun$prettyIndex$1.apply(ModelGroup.scala:147)
>  at 
> org.apache.daffodil.dsom.ModelGroup$$anonfun$prettyIndex$1.apply(ModelGroup.scala:152)
>  at org.apache.daffodil.oolag.OOLAG$OOLAGValue.liftedTree1$1(OOLAG.scala:585)
>  at 
> org.apache.daffodil.oolag.OOLAG$OOLAGValue.value$lzycompute(OOLAG.scala:583)
>  at org.apache.daffodil.oolag.OOLAG$OOLAGValue.value(OOLAG.scala:581)
>  at org.apache.daffodil.dsom.ModelGroup.prettyIndex(ModelGroup.scala:152)
>  at 
> org.apache.daffodil.dsom.ModelGroup.diagnosticDebugName$lzycompute(ModelGroup.scala:155)
>  at 
> org.apache.daffodil.dsom.ModelGroup.diagnosticDebugName(ModelGroup.scala:155)
>  at 
> org.apache.daffodil.dsom.SchemaComponent$class.toString(SchemaComponent.scala:180)
>  at org.apache.daffodil.dsom.ModelGroup.toString(ModelGroup.scala:129)
>  at 
> org.apache.daffodil.exceptions.SchemaFileLocation.(SchemaFileLocatable.scala:52)
>  at 
> org.apache.daffodil.exceptions.SchemaFileLocatable$class.schemaFileLocation(SchemaFileLocatable.scala:131)
>  at 
> org.apache.daffodil.dsom.ModelGroup.schemaFileLocation$lzycompute(ModelGroup.scala:129)
>  at 
> org.apache.daffodil.dsom.ModelGroup.schemaFileLocation(ModelGroup.scala:129)
>  at 
> org.apache.daffodil.dsom.ImplementsThrowsOrSavesSDE$class.SDW(SDE.scala:190)
>  at org.apache.daffodil.dsom.ModelGroup.SDW(ModelGroup.scala:129)
>  at 
> org.apache.daffodil.dsom.AnnotatedMixin$$anonfun$11.apply(AnnotatedSchemaComponent.scala:288)
>  at 
> org.apache.daffodil.dsom.AnnotatedMixin$$anonfun$11.apply(AnnotatedSchemaComponent.scala:284)
>  at 
> scala.collection.TraversableLike$$anonfun$filterImpl$1.apply(TraversableLike.scala:248)
>  at scala.collection.Iterator$class.foreach(Iterator.scala:893)
>  at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
>  at 

[GitHub] jadams-tresys commented on a change in pull request #164: Avoid CircularDefinition error for appinfo

[GitBox]

jadams-tresys commented on a change in pull request #164: Avoid 
CircularDefinition error for appinfo
URL: https://github.com/apache/incubator-daffodil/pull/164#discussion_r247627076
 
 

 ##
 File path: 
daffodil-core/src/main/scala/org/apache/daffodil/dsom/ModelGroup.scala
 ##
 @@ -153,14 +153,7 @@ abstract class ModelGroup
   final override def isRequiredOrComputed = true
   final override def isArray = false
 
-  private def prettyIndex = LV('prettyIndex) {
-myPeers.map { peers =>
-  {
-if (peers.length == 1) "" // no index expression if we are the only one
-else "[" + (peers.indexOf(this) + 1) + "]" // 1-based indexing in 
XML/XSD
-  }
-}.getOrElse("")
-  }.value
+  private def prettyIndex = "[" + index + "]" // 1-based indexing in XML/XSD
 
 Review comment:
   Fixed this issue by having it look for seq[1] instead of just seq.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] jadams-tresys merged pull request #164: Avoid CircularDefinition error for appinfo

[GitBox]

jadams-tresys merged pull request #164: Avoid CircularDefinition error for 
appinfo
URL: https://github.com/apache/incubator-daffodil/pull/164
 
 
   

As this is a foreign pull request (from a fork), the diff has been
sent to your commit mailing list, comm...@daffodil.apache.org


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Updated] (DAFFODIL-1784) Modify Bamboo to trigger a Daffodil NiFi Processor build when daffodil changes

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-1784:
-
Fix Version/s: 2.3.0

> Modify Bamboo to trigger a Daffodil NiFi Processor build when daffodil changes
> --
>
> Key: DAFFODIL-1784
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1784
> Project: Daffodil
>  Issue Type: Task
>  Components: Infrastructure
>Reporter: Steve Lawrence
>Priority: Major
> Fix For: 2.3.0
>
>
> Things we'll need to modify
>  * Modify Daffodil build to run 'sbt publishM2' to publish the Daffodil 
> snapshot to a local maven repo (or 'sbt publish/publishSigned') to publish to 
> the public nexus server
>  * Add a trigger to build the nar repo (make sure maven update looks to 
> update snapshots, -U option should work?)
>  * Talk to Rob about getting the nar added as an artifact



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DAFFODIL-1833) TDML runner - way to disable multi-round-trip cycle

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-1833:
-
Fix Version/s: 2.3.0

> TDML runner - way to disable multi-round-trip cycle
> ---
>
> Key: DAFFODIL-1833
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1833
> Project: Daffodil
>  Issue Type: New Feature
>  Components: Infrastructure, TDML Runner
>Reporter: Michael Beckerle
>Priority: Major
> Fix For: 2.3.0
>
>
> For round-trip tests, the TDML runner tries parse-unparse, then if it doesn't 
> compare it does another parse and unparse.
> There needs to be an option to turn this off on a per test case basis so that 
> one can create a test and insist that the round trip work in exactly one 
> cycle of parse and unparse, or for an unparserTestCase unparse-and-parse.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DAFFODIL-2044) Unexpected error on malformed config file

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-2044:
-
Fix Version/s: 2.3.0

> Unexpected error on malformed config file
> -
>
> Key: DAFFODIL-2044
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2044
> Project: Daffodil
>  Issue Type: Bug
>  Components: CLI
>Affects Versions: 2.1.0
>Reporter: Brandon Sloane
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.3.0
>
>
> When daffodil is run with a malformed config file, it reports an unexpected 
> excpetion.
> For instance, on the config:
> {quote}
> http://www.w3.org/2001/XMLSchema;>
>  
>  x
>  
> 
> {quote}
> It outputs:
> {quote}!!
> !! An unexpected exception occurred. This is a bug! !!
> !!
> Please report this bug and help us fix it:
> https://daffodil.apache.org/community/#issue-tracker
> Please include the following exception, the command you
>  ran, and any input, schema, or tdml files used that led
>  to this bug.
> org.apache.daffodil.xml.QNameUndefinedPrefixException: Undefined QName prefix 
> 'a'
>  at 
> org.apache.daffodil.xml.RefQNameFactoryBase$$anonfun$resolveRef$1.apply(QNameBase.scala:481)
>  at scala.util.Try$.apply(Try.scala:192)
>  at 
> org.apache.daffodil.xml.RefQNameFactoryBase$class.resolveRef(QNameBase.scala:471)
>  at org.apache.daffodil.xml.RefQNameFactory$.resolveRef(QNameBase.scala:492)
>  at org.apache.daffodil.xml.QName$.resolveRef(QNameBase.scala:111)
>  at org.apache.daffodil.externalvars.Binding$.apply(Binding.scala:64)
>  at 
> org.apache.daffodil.externalvars.ExternalVariablesLoader$$anonfun$getBindings$1.apply(ExternalVariablesLoader.scala:130)
>  at 
> org.apache.daffodil.externalvars.ExternalVariablesLoader$$anonfun$getBindings$1.apply(ExternalVariablesLoader.scala:130)
>  at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>  at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>  at scala.collection.Iterator$class.foreach(Iterator.scala:893)
>  at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
>  at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>  at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>  at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>  at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>  at 
> org.apache.daffodil.externalvars.ExternalVariablesLoader$.getBindings(ExternalVariablesLoader.scala:130)
>  at 
> org.apache.daffodil.externalvars.ExternalVariablesLoader$.getVariables(ExternalVariablesLoader.scala:60)
>  at org.apache.daffodil.Main$.retrieveExternalVariables(Main.scala:627)
>  at org.apache.daffodil.Main$.run(Main.scala:821)
>  at org.apache.daffodil.Main$.main(Main.scala:1297)
>  at org.apache.daffodil.Main.main(Main.scala)
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DAFFODIL-1427) Use scala 2.11 options to clean up dead-code, unused imports, unused methods, unused local vals, etc.

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-1427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-1427:
-
Fix Version/s: 2.3.0

> Use scala 2.11 options to clean up dead-code, unused imports, unused methods, 
> unused local vals, etc.
> -
>
> Key: DAFFODIL-1427
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1427
> Project: Daffodil
>  Issue Type: Improvement
>  Components: Infrastructure
>Reporter: Michael Beckerle
>Assignee: Michael Beckerle
>Priority: Major
> Fix For: 2.3.0
>
>
> Scala 2.11 supports
> -Ywarn-dead-code  -Ywarn-inaccessible -Ywarn-unused-import -Ywarn-unused 
> -Ywarn-infer-any -Ywarn-nullary-override
> These will generate warnings that allow us to clean up the code base 
> substantially, and may catch bugs - where we have unused local values for 
> example, but thought we were using them but aren't.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DAFFODIL-1324) Unparse: Sequences need way to skip over optional branches

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-1324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-1324:
-
Fix Version/s: 2.3.0

> Unparse: Sequences need way to skip over optional branches
> --
>
> Key: DAFFODIL-1324
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1324
> Project: Daffodil
>  Issue Type: Bug
>  Components: Unparsing
>Reporter: Steve Lawrence
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.3.0
>
>
> For example, if we have something like this:
> {code:xml}
> 
>   
>   
> 
> {code}
> The unparser generates looks something like this (not that optional elements 
> are treated as arrays):
> {code:xml}
> 
>...
>...
> 
> {code}
> When the SequenceCombinatorUnparser gets an event it could the Start(DIArray) 
> of either of the elements, since the first one is optional. I think we need 
> something similar to the ChoiceMap where based on the next element received, 
> it can skip over some of the parsers. This avoids having to do any 
> backtracking kindof work.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DAFFODIL-728) Determine requirements for Java API (unparsing)

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-728:

Fix Version/s: 2.3.0

> Determine requirements for Java API (unparsing)
> ---
>
> Key: DAFFODIL-728
> URL: https://issues.apache.org/jira/browse/DAFFODIL-728
> Project: Daffodil
>  Issue Type: Task
>Reporter: Jessie Chab
>Assignee: Michael Beckerle
>Priority: Major
> Fix For: 2.3.0
>
>
> define requirements for unparsing aspects of Java API



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DAFFODIL-1441) Implement examples for DFDLSchemas

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-1441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-1441:
-
Fix Version/s: 2.3.0

> Implement examples for DFDLSchemas
> --
>
> Key: DAFFODIL-1441
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1441
> Project: Daffodil
>  Issue Type: New Feature
>  Components: Infrastructure
>Reporter: Michael Beckerle
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.3.0
>
>
> Several schemas have been published on github DFDLSchemas.
> We should extend our examples to include these. If they won't work yet, then 
> we can leave the tests that drive them in scala-debug.
> NACHA
> EDIFACT
> vCard
> IBM4690-TLOG,
> HL7
> Note that the SWIFT DFDL Schema isn't published at github directly. We would 
> have to ask for a copy and permission from IBM to test this one.
> Note that we should NOT duplicate these in our tree, but rather should have 
> the examples expect them to reside in some location relative to DAFFODIL_HOME.
> Also: Any edits to these schemas should be done on git branches to isolate 
> any daffodil-specific changes - until shown that the schemas continue to work 
> on IBM's implementation with those changes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DAFFODIL-916) StackOverflowError when attempting to parse >=200MB CSV

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-916:

Fix Version/s: 2.3.0

> StackOverflowError when attempting to parse >=200MB CSV
> ---
>
> Key: DAFFODIL-916
> URL: https://issues.apache.org/jira/browse/DAFFODIL-916
> Project: Daffodil
>  Issue Type: Bug
>Reporter: Jessie Chab
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.3.0
>
>
> I have been creating data and tests for our "csvMixedNarrow" performance set. 
> When attempting to parse the 200MB, 250MB, 500MB, and 1GB files, I get the 
> following errors (both with tdml runner and CLI, giving sbt/start script 8GB 
> of memory):
> {noformat}
> [jchab@localhost dfdl]$ ./daffodil-cli/target/start parse -s 
> ../ngf-dfdl/daffodil-perf/src/test/resources/edu/illinois/ncsa/daffodil/csvMixedNarrow/csvMixedNarrow.dfdl.xsd
>  
> ../ngf-dfdl/daffodil-perf/src/test/resources/edu/illinois/ncsa/daffodil/csvMixedNarrow/peopleData_1g.txt
>  
> Exception in thread "main" java.lang.StackOverflowError
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at scala.collection.immutable.Page.latest(PagedSeq.scala:237)
>   at 

[jira] [Updated] (DAFFODIL-1335) mil-std-2045 should assert that data is not compressed

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-1335:
-
Fix Version/s: 2.3.0

> mil-std-2045 should assert that data is not compressed
> --
>
> Key: DAFFODIL-1335
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1335
> Project: Daffodil
>  Issue Type: Bug
>  Components: DFDL Schemas
>Affects Versions: 1.1.0
>Reporter: Jessie Chab
>Priority: Major
> Fix For: 2.3.0
>
>
> There seems to be an issue where encoding is not handled properly when 
> parsing strings. All numeric data seems to be parsing correctly, but any 
> parsed strings appear to be garbage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DAFFODIL-1221) Abort: Invariant Broken when XPath expression moves past root

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-1221:
-
Fix Version/s: 2.3.0

> Abort: Invariant Broken when XPath expression moves past root
> -
>
> Key: DAFFODIL-1221
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1221
> Project: Daffodil
>  Issue Type: Bug
>  Components: General
>Affects Versions: 1.0.0
>Reporter: Jessie Chab
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.3.0
>
>
> If the XPath expression extends beyond the root of the tree, you get the 
> following error:
> {code}
> [error] Test 
> edu.illinois.ncsa.daffodil.section23.dfdl_expressions.TestDFDLExpressions.test_beyondRoot_01
>  failed: edu.illinois.ncsa.daffodil.exceptions.Abort: Invariant broken. 
> UpMove past root. Should never happen since an expression like that won't 
> typecheck statically.
> [error] edu.illinois.ncsa.daffodil.exceptions.Assert$.abort(Assert.scala:95)
> [error] 
> edu.illinois.ncsa.daffodil.exceptions.Assert$.invariantFailed(Assert.scala:136)
> [error] 
> edu.illinois.ncsa.daffodil.dpath.UpMove$$anonfun$1.apply(UpDownMoves.scala:32)
> [error] 
> edu.illinois.ncsa.daffodil.dpath.UpMove$$anonfun$1.apply(UpDownMoves.scala:32)
> [error] at 
> edu.illinois.ncsa.daffodil.exceptions.Assert$.abort(Assert.scala:95)
> [error] at 
> edu.illinois.ncsa.daffodil.exceptions.Assert$.invariantFailed(Assert.scala:136)
> [error] at 
> edu.illinois.ncsa.daffodil.dpath.UpMove$$anonfun$1.apply(UpDownMoves.scala:32)
> [error] at 
> edu.illinois.ncsa.daffodil.dpath.UpMove$$anonfun$1.apply(UpDownMoves.scala:32)
> [error] at 
> edu.illinois.ncsa.daffodil.util.Maybe$.getOrElse$extension(Maybe.scala:30)
> [error] at 
> edu.illinois.ncsa.daffodil.dpath.UpMove$.run(UpDownMoves.scala:32)
> [error] at 
> edu.illinois.ncsa.daffodil.dpath.CompiledDPath.run(DPathRuntime.scala:85)
> [error] at 
> edu.illinois.ncsa.daffodil.dpath.CompiledDPath.runExpression(DPathRuntime.scala:31)
> [error] at 
> edu.illinois.ncsa.daffodil.dpath.RuntimeExpressionDPath.liftedTree1$1(DPath.scala:92)
> [error] at 
> edu.illinois.ncsa.daffodil.dpath.RuntimeExpressionDPath.evaluate(DPath.scala:91)
> [error] at 
> edu.illinois.ncsa.daffodil.processors.parsers.ExpressionEvaluationParser.eval(ExpressionEvaluatingParsers.scala:30)
> [error] at 
> edu.illinois.ncsa.daffodil.processors.parsers.IVCParser$$anonfun$parse$1.apply(ExpressionEvaluatingParsers.scala:44)
> [error] at 
> edu.illinois.ncsa.daffodil.processors.parsers.IVCParser$$anonfun$parse$1.apply(ExpressionEvaluatingParsers.scala:41)
> [error] at 
> edu.illinois.ncsa.daffodil.processors.WithParseErrorThrowing$class.withParseErrorThrowing(ParseErrors.scala:205)
> [error] at 
> edu.illinois.ncsa.daffodil.processors.parsers.ExpressionEvaluationParser.withParseErrorThrowing(ExpressionEvaluatingParsers.scala:19)
> [error] at 
> edu.illinois.ncsa.daffodil.processors.parsers.IVCParser.parse(ExpressionEvaluatingParsers.scala:41)
> 
> {code}
> I first noticed this with the interactive debugger, but have written a TDML 
> test to demonstrate the issue:
> test_beyondRoot_01
> daffodil-test/src/test/resources/edu/illinois/ncsa/daffodil/section23/dfdl_expressions/expressions.tdml
> daffodil-test/src/test/scala-debug/edu/illinois/ncsa/daffodil/section23/dfdl_expressions/TestDFDLExpressionsDebug.scala
> daffodil-test/src/test/scala/edu/illinois/ncsa/daffodil/section23/dfdl_expressions/TestDFDLExpressions.scala



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DAFFODIL-711) Hefty Abort Message when using an expression (and previous elements) to set formatting properties

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-711:

Fix Version/s: 2.3.0

> Hefty Abort Message when using an expression (and previous elements) to set 
> formatting properties
> -
>
> Key: DAFFODIL-711
> URL: https://issues.apache.org/jira/browse/DAFFODIL-711
> Project: Daffodil
>  Issue Type: Bug
>  Components: Back End, Diagnostics, Front End, Usability
>Reporter: Jessie Chab
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.3.0
>
>
> I'm not sure if this functionality is even implemented yet but I still don't 
> believe I should be getting such a vague message, especially an Abort.
> 
> [error] Test 
> edu.illinois.ncsa.daffodil.section07.variables.TestVariables.test_setVarAbsolutePath
>  {color:red}failed: edu.illinois.ncsa.daffodil.exceptions.Abort: Boolean 
> isConstant is false. Cannot request a constant value.{color}
> [error] edu.illinois.ncsa.daffodil.exceptions.Assert$.abort(Assert.scala:87)
> [error] 
> edu.illinois.ncsa.daffodil.exceptions.Assert$.usageError(Assert.scala:75)
> [error] 
> edu.illinois.ncsa.daffodil.dsom.RuntimeExpression.constant(CompiledExpression.scala:131)
> [error] 
> edu.illinois.ncsa.daffodil.dsom.CompiledExpression.constantAsString(CompiledExpression.scala:97)
> [error] at 
> edu.illinois.ncsa.daffodil.exceptions.Assert$.abort(Assert.scala:87)
> [error] at 
> edu.illinois.ncsa.daffodil.exceptions.Assert$.usageError(Assert.scala:75)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.RuntimeExpression.constant(CompiledExpression.scala:131)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.CompiledExpression.constantAsString(CompiledExpression.scala:97)
> [error] at 
> edu.illinois.ncsa.daffodil.processors.StaticText$$anonfun$11.apply(PrimitivesDelimiters.scala:81)
> [error] at 
> edu.illinois.ncsa.daffodil.processors.StaticText$$anonfun$11.apply(PrimitivesDelimiters.scala:80)
> [error] at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> [error] at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> [error] at scala.collection.immutable.List.foreach(List.scala:318)
> [error] at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> [error] at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> [error] at 
> edu.illinois.ncsa.daffodil.processors.StaticText.(PrimitivesDelimiters.scala:80)
> [error] at 
> edu.illinois.ncsa.daffodil.processors.StaticDelimiter.(PrimitivesDelimiters.scala:65)
> [error] at 
> edu.illinois.ncsa.daffodil.processors.StaticSeparator.(PrimitivesDelimiters.scala:453)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.TermGrammarMixin$$anonfun$staticSeparator$1.apply(GrammarMixins.scala:1080)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.TermGrammarMixin$$anonfun$staticSeparator$1.apply(GrammarMixins.scala:1080)
> [error] at 
> edu.illinois.ncsa.daffodil.grammar.Prod$$anonfun$26.apply(Grammar.scala:296)
> [error] at 
> edu.illinois.ncsa.daffodil.grammar.Prod$$anonfun$26.apply(Grammar.scala:294)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.oolag.OOLAG$LV.lazyBody$lzycompute(OOLAG.scala:674)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.oolag.OOLAG$LV.lazyBody(OOLAG.scala:674)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.oolag.OOLAG$OOLAGValue$$anonfun$6$$anonfun$apply$21.apply(OOLAG.scala:591)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.oolag.OOLAG$OOLAGValue.edu$illinois$ncsa$daffodil$dsom$oolag$OOLAG$OOLAGValue$$oolagTryCatch(OOLAG.scala:498)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.oolag.OOLAG$OOLAGValue$$anonfun$6.apply(OOLAG.scala:595)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.oolag.OOLAG$OOLAGHost.circularityDetector(OOLAG.scala:236)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.oolag.OOLAG$OOLAGValue.valueAsAny(OOLAG.scala:581)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.oolag.OOLAG$LV.value(OOLAG.scala:677)
> [error] at 
> edu.illinois.ncsa.daffodil.grammar.Prod.gram$lzycompute(Grammar.scala:292)
> [error] at edu.illinois.ncsa.daffodil.grammar.Prod.gram(Grammar.scala:292)
> [error] at 
> edu.illinois.ncsa.daffodil.grammar.Prod.deref(Grammar.scala:277)
> [error] at edu.illinois.ncsa.daffodil.grammar.Gram.$bar(Grammar.scala:89)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.TermGrammarMixin$class.sepRule(GrammarMixins.scala:1085)
> [error] at 
> edu.illinois.ncsa.daffodil.dsom.Term.sepRule$lzycompute(Groups.scala:53)
> [error] at edu.illinois.ncsa.daffodil.dsom.Term.sepRule(Groups.scala:53)
> [error] at 
> 

[jira] [Updated] (DAFFODIL-2020) Infinite loop if the dfdl:hiddenGroupRef reference does not exist

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-2020:
-
Fix Version/s: 2.3.0

> Infinite loop if the dfdl:hiddenGroupRef reference does not exist
> -
>
> Key: DAFFODIL-2020
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2020
> Project: Daffodil
>  Issue Type: Bug
>  Components: Front End
>Affects Versions: 2.2.0
>Reporter: Steve Lawrence
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.3.0
>
>
> Daffodil gets stuck in what appears to be an infinite loop if you have an 
> {{xs:sequence}} that defines a {{dfdl:hiddenGroupRef}} where the reference 
> does not resolve to anything, e.g.:
> {code:xml}
> ...
> 
> ...{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DAFFODIL-2031) Type with xs:hexBinary requires the dfdl:binaryNumberRep property

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-2031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-2031:
-
Fix Version/s: 2.3.0

> Type with xs:hexBinary requires the dfdl:binaryNumberRep property
> -
>
> Key: DAFFODIL-2031
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2031
> Project: Daffodil
>  Issue Type: Bug
>  Components: Front End
>Affects Versions: 2.3.0
>Reporter: Steve Lawrence
>Assignee: Dave Thompson
>Priority: Major
> Fix For: 2.3.0
>
>
> If a simple type is specified as xs:hexBinary, we currently get an SDE if the 
> dfdl:binaryNumberRep property is not defined. That property does not a apply 
> to hexBInary types and should not be required.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (DAFFODIL-1946) Dates with out of range years will overflow to a valid value

[Steve Lawrence (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Lawrence updated DAFFODIL-1946:
-
Fix Version/s: 2.3.0

> Dates with out of range years will overflow to a valid value
> 
>
> Key: DAFFODIL-1946
> URL: https://issues.apache.org/jira/browse/DAFFODIL-1946
> Project: Daffodil
>  Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Elizabeth Fahl
>Assignee: Dave Thompson
>Priority: Minor
> Fix For: 2.3.0
>
>
> When trying to parse an invalid date, due to being out of the valid range for 
> Calendar, currently the date overflows to a valid date, but then it is 
> incorrect. This should be a failure instead. This affects both text and 
> binary representation.
> See test_dateTimeImplicitPatternFail5 in TestSimpleTypesDebug.Scala



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] stevedlawrence opened a new pull request #165: Update README to require Java 8+

[GitBox]

stevedlawrence opened a new pull request #165: Update README to require Java 8+
URL: https://github.com/apache/incubator-daffodil/pull/165
 
 
   DAFFODIL-2025


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] stevedlawrence commented on a change in pull request #164: Avoid CircularDefinition error for appinfo

[GitBox]

stevedlawrence commented on a change in pull request #164: Avoid 
CircularDefinition error for appinfo
URL: https://github.com/apache/incubator-daffodil/pull/164#discussion_r247589175
 
 

 ##
 File path: 
daffodil-core/src/main/scala/org/apache/daffodil/dsom/ModelGroup.scala
 ##
 @@ -153,14 +153,7 @@ abstract class ModelGroup
   final override def isRequiredOrComputed = true
   final override def isArray = false
 
-  private def prettyIndex = LV('prettyIndex) {
-myPeers.map { peers =>
-  {
-if (peers.length == 1) "" // no index expression if we are the only one
-else "[" + (peers.indexOf(this) + 1) + "]" // 1-based indexing in 
XML/XSD
-  }
-}.getOrElse("")
-  }.value
+  private def prettyIndex = "[" + index + "]" // 1-based indexing in XML/XSD
 
 Review comment:
   Looks like this change caused some CLI tests to fail. I'd guess because we 
now always output "[X]" even if there are no peers. I think this change is fine 
since this is more a debug thing, so always outputting the index doesn't 
matter, but we should update the CLI tests so they all pass. To runt he CLI 
tests, you can run  ``sbt it:test``.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Assigned] (DAFFODIL-2050) Line numbers missing in schema definition errors

[Josh Adams (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DAFFODIL-2050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Adams reassigned DAFFODIL-2050:


Assignee: Josh Adams

> Line numbers missing in schema definition errors
> 
>
> Key: DAFFODIL-2050
> URL: https://issues.apache.org/jira/browse/DAFFODIL-2050
> Project: Daffodil
>  Issue Type: Bug
>  Components: Diagnostics, Front End
>Reporter: Steve Lawrence
>Assignee: Josh Adams
>Priority: Critical
> Fix For: 2.3.0
>
>
> Commit 55ddb637732fca7cabc79258c2434d516a87993f created a regression that 
> caused schema definition errors to no longer include line numbers. Need to 
> determine what caused the regression and fix.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Things I noticed

[Steve Lawrence]

Can you post the schema and example input file?

On 1/14/19 8:19 AM, Christofer Dutz wrote:
> Being an Apache Committer you can even get a free license of IntelliJ 
> Utlimate from Jetbrains for working on Open-Source projects.
> 
> Even when using the commit before the line-number regression, it still didn't 
> show me any line numbers ... so I went back to the 2.2.0-rc2 tag and that got 
> things working.
> 
> Now I've got my schema sort of correct and try to read some binary data. 
> Unfortunately the CLI seems to think it has 0 bytes to read. I debugged into 
> the InputSource creation and the FileInputStream tells me it's got 12 bytes, 
> but Daffodil fails reading the first byte complaining it's got 0 bits to 
> read. Anything I should know?
> 
> Chris
> 
> 
> 
> Am 14.01.19, 13:24 schrieb "Steve Lawrence" :
> 
> The README just needs to be updated. Java 8 is required.
> 
> Good to know about debugging with IntelliJ. I believe most daffodil devs
> either use sbt with something like vim and printf style debugging, or
> eclipse. I'll have to take a look at IntelliJ.
> 
> - Steve
> 
> On 1/14/19 6:30 AM, Christofer Dutz wrote:
> > Hi all,
> > 
> > As I am expecting to be participating here in the near future I started 
> tracking down some problems I was having with debugging the CLI … Here’s a 
> summary of things I noticed:
> > 
> > General Observations:
> > The Readme claims to require Java 7 or above … I think it requires 8 … 
> even if the build does sort of run with Java 7, as soon as I execute anything 
> I am getting “Unsupported major.minor version 52.0” errors.
> > I guess some library might be referenced that requires Java 8.
> > 
> > Debugging in IntelliJ:
> > Even if I am able to run things in IntelliJ, when it comes to debugging 
> it’s almost not usable as I can’t inspect any variables.
> > All I can see is the names and a “Collecting data …” behind them.
> > 
> > So I consulted a Scala expert here at my company and he suggested to 
> manually install sbt (Not use the bundled sbt-launcher contained in the 
> IntelliJ SBT plugin) and start it on a console with:
> > 
> > sbt -jvm-debug 5005 -mem 8123
> > 
> > The I connected to that via normal IntelliJ Java remote debugging 
> connection.
> > 
> > After that as long as I don’t expand the “this” variable if it says 
> anything with “ROOT” in it, I am able to debug things.
> > 
> > Chris
> > 
> > 
> 
> 
> 

Re: Things I noticed

[Christofer Dutz]

Being an Apache Committer you can even get a free license of IntelliJ Utlimate 
from Jetbrains for working on Open-Source projects.

Even when using the commit before the line-number regression, it still didn't 
show me any line numbers ... so I went back to the 2.2.0-rc2 tag and that got 
things working.

Now I've got my schema sort of correct and try to read some binary data. 
Unfortunately the CLI seems to think it has 0 bytes to read. I debugged into 
the InputSource creation and the FileInputStream tells me it's got 12 bytes, 
but Daffodil fails reading the first byte complaining it's got 0 bits to read. 
Anything I should know?

Chris



Am 14.01.19, 13:24 schrieb "Steve Lawrence" :

The README just needs to be updated. Java 8 is required.

Good to know about debugging with IntelliJ. I believe most daffodil devs
either use sbt with something like vim and printf style debugging, or
eclipse. I'll have to take a look at IntelliJ.

- Steve

On 1/14/19 6:30 AM, Christofer Dutz wrote:
> Hi all,
> 
> As I am expecting to be participating here in the near future I started 
tracking down some problems I was having with debugging the CLI … Here’s a 
summary of things I noticed:
> 
> General Observations:
> The Readme claims to require Java 7 or above … I think it requires 8 … 
even if the build does sort of run with Java 7, as soon as I execute anything I 
am getting “Unsupported major.minor version 52.0” errors.
> I guess some library might be referenced that requires Java 8.
> 
> Debugging in IntelliJ:
> Even if I am able to run things in IntelliJ, when it comes to debugging 
it’s almost not usable as I can’t inspect any variables.
> All I can see is the names and a “Collecting data …” behind them.
> 
> So I consulted a Scala expert here at my company and he suggested to 
manually install sbt (Not use the bundled sbt-launcher contained in the 
IntelliJ SBT plugin) and start it on a console with:
> 
> sbt -jvm-debug 5005 -mem 8123
> 
> The I connected to that via normal IntelliJ Java remote debugging 
connection.
> 
> After that as long as I don’t expand the “this” variable if it says 
anything with “ROOT” in it, I am able to debug things.
> 
> Chris
> 
> 

Re: Things I noticed

[Steve Lawrence]

The README just needs to be updated. Java 8 is required.

Good to know about debugging with IntelliJ. I believe most daffodil devs
either use sbt with something like vim and printf style debugging, or
eclipse. I'll have to take a look at IntelliJ.

- Steve

On 1/14/19 6:30 AM, Christofer Dutz wrote:
> Hi all,
> 
> As I am expecting to be participating here in the near future I started 
> tracking down some problems I was having with debugging the CLI … Here’s a 
> summary of things I noticed:
> 
> General Observations:
> The Readme claims to require Java 7 or above … I think it requires 8 … even 
> if the build does sort of run with Java 7, as soon as I execute anything I am 
> getting “Unsupported major.minor version 52.0” errors.
> I guess some library might be referenced that requires Java 8.
> 
> Debugging in IntelliJ:
> Even if I am able to run things in IntelliJ, when it comes to debugging it’s 
> almost not usable as I can’t inspect any variables.
> All I can see is the names and a “Collecting data …” behind them.
> 
> So I consulted a Scala expert here at my company and he suggested to manually 
> install sbt (Not use the bundled sbt-launcher contained in the IntelliJ SBT 
> plugin) and start it on a console with:
> 
> sbt -jvm-debug 5005 -mem 8123
> 
> The I connected to that via normal IntelliJ Java remote debugging connection.
> 
> After that as long as I don’t expand the “this” variable if it says anything 
> with “ROOT” in it, I am able to debug things.
> 
> Chris
> 
> 

Things I noticed

[Christofer Dutz]

Hi all,

As I am expecting to be participating here in the near future I started 
tracking down some problems I was having with debugging the CLI … Here’s a 
summary of things I noticed:

General Observations:
The Readme claims to require Java 7 or above … I think it requires 8 … even if 
the build does sort of run with Java 7, as soon as I execute anything I am 
getting “Unsupported major.minor version 52.0” errors.
I guess some library might be referenced that requires Java 8.

Debugging in IntelliJ:
Even if I am able to run things in IntelliJ, when it comes to debugging it’s 
almost not usable as I can’t inspect any variables.
All I can see is the names and a “Collecting data …” behind them.

So I consulted a Scala expert here at my company and he suggested to manually 
install sbt (Not use the bundled sbt-launcher contained in the IntelliJ SBT 
plugin) and start it on a console with:

sbt -jvm-debug 5005 -mem 8123

The I connected to that via normal IntelliJ Java remote debugging connection.

After that as long as I don’t expand the “this” variable if it says anything 
with “ROOT” in it, I am able to debug things.

Chris