[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-30 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r184990093
 
 

 ##
 File path: 
daffodil-runtime1/src/main/scala/org/apache/daffodil/infoset/ScalaXMLInfosetInputter.scala
 ##
 @@ -81,6 +82,21 @@ class ScalaXMLInfosetInputter(rootNode: Node)
 }
 val text = child match {
   case t: Text => t.data
+  //
+  // Note: may be a bug in scala.xml library, but if you do this:
 
 Review comment:
   I have to revise this comment. The comment code here doesn't even work. I 
did have to add Atom[String] case, and I'm not sure why that should ever occur, 
but the example for reproducing it here doesn't work. There's at least one 
other place in XMLUtils where comments on XML wierdness about Text vs. PCData 
appear, and when I re-run those examples, they don't have the problems 
described in the comments anymore. I''ll fix the comment here, and update that 
other place to point out that the problem no longer occurs. Not sure what else 
I can do about this other than revisit all the use of XML parsers in Daffodil? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-30 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r184977533
 
 

 ##
 File path: 
daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaIinputStreams.scala
 ##
 @@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.io
+
+import org.apache.daffodil.exceptions.Assert
+import java.util.regex.Pattern
+import java.util.Scanner
+import java.nio.charset.Charset
+import java.io.InputStream
+import java.io.FilterInputStream
+import java.io.InputStreamReader
+import java.nio.charset.StandardCharsets
+
+/**
+ * This class can be used with any InputStream to restrict what is
+ * read from it to N bytes.
+ *
+ * This can be used to forcibly stop consumption of data from a stream at
+ * a length obtained explicitly.
+ *
+ * Thread safety: This is inherently stateful - so not thread safe to use
+ * this object from more than one thread.
+ */
+class ExplicitLengthLimitingStream(in: InputStream, limit: Int)
+  extends FilterInputStream(in) {
+
+  private var numRemaining = limit
+
+  override def read(buf: Array[Byte], off: Int, len: Int) = {
+Assert.invariant(numRemaining >= 0)
+if (numRemaining == 0) -1
+else if (len == 0) 0
+else {
+  val requestSize = math.min(numRemaining, len)
+  val actualSize = in.read(buf, off, requestSize)
+  if (actualSize == -1)
+numRemaining = 0
+  else
+numRemaining -= actualSize
+  actualSize
+}
+  }
+
+  private val readBuf = new Array[Byte](1)
+
+  override def read(): Int = {
+readBuf(0) = 0
+val n = read(readBuf, 0, 1)
+if (n == -1)
+  -1
+else {
+  Assert.invariant(n == 1)
+  val i = readBuf(0).toInt
+  val b = i & 0xFF
+  b
+}
+  }
+}
+
+/**
+ * Can be used with any InputStream to restrict what is
+ * read from it to stop before a boundary mark string.
+ *
+ * The boundary mark string is exactly that, a string of characters. Not a
+ * regex, nor anything involving DFDL Character Entities or Character Class
+ * Entities. (No %WSP; no %NL; )
+ *
+ * This can be used to forcibly stop consumption of data from a stream at
+ * a length obtained from a delimiter.
+ *
+ * The boundary mark string is consumed from the underlying stream (if found), 
and
+ * the underlying stream is left positioned at the byte after the boundary mark
+ * string.
+ *
+ * Thread safety: This is inherently stateful - so not thread safe to use
+ * this object from more than one thread.
+ */
+object BoundaryMarkLimitingStream {
+
+  def apply(inputStream: InputStream, boundaryMark: String, charset: Charset,
+targetChunkSize: Int = 32 * 1024) = {
+
+Assert.usage(targetChunkSize >= 1)
+Assert.usage(boundaryMark.length >= 1)
+
+val boundaryMarkIn8859 = new String(boundaryMark.getBytes(charset), 
StandardCharsets.ISO_8859_1)
+
+val quotedBoundaryMark = Pattern.quote(boundaryMarkIn8859) // in case 
pattern has non-regex-safe characters in it
+
+val result = new RegexLimitingStream(inputStream, quotedBoundaryMark, 
boundaryMarkIn8859, charset, targetChunkSize)
+result
+  }
+
+}
+
+class StreamIterator[T](s: Stream[T])
+  extends Iterator[T] {
+  private var str = s
+  override def hasNext = !str.isEmpty
+  override def next() = {
+val res = str.head
+str = str.tail
+res
+  }
+}
+
+/**
+ * Can be used with any InputStream to restrict what is
+ * read from it to stop before a particular regex match.
+ *
+ * The regex must have a finite maximum length match string.
+ *
+ * This can be used to forcibly stop consumption of data from a stream at
+ * a length obtained from a delimiter that is described using a regex.
+ *
+ * The delimiter matching the regex is consumed from the underlying stream (if 
found), and
+ * the underlying stream is left positioned at the byte after the regex match
+ * string.
+ *
+ * IMPORTANT: The delimiter regex cannot contain any Capturing Groups!
 
 Review comment:
   Hi Taylor, no nothing checks this as yet. It is possible to check, as regex 
are checkable that way, but it's non-trivial.  I'm thinking a 

[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-20 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r183101705
 
 

 ##
 File path: 
daffodil-core/src/test/scala/org/apache/daffodil/layers/TestLayers.scala
 ##
 @@ -38,7 +38,7 @@ class TestLayers {
   ,
   
 
-  
+  
 
 Review comment:
   Yes, I removed the discussion of parent delimiting markup having impact. 
   
   I like changing the layerLengthKind to boundaryMark also. Will make that 
change. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-20 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r183098708
 
 

 ##
 File path: 
daffodil-io/src/test/scala/org/apache/daffodil/layers/TestBase64.scala
 ##
 @@ -36,16 +36,25 @@ import org.apache.daffodil.exceptions.Assert
 class TestBase64 {
   Assert.usage(scala.util.Properties.isJavaAtLeast("1.8"))
 
-  val text = """Man is distinguished, not only by his reason, but by this 
singular passion from
-other animals, which is a lust of the mind, that by a perseverance of delight
-in the continued and indefatigable generation of knowledge, exceeds the short
-vehemence of any carnal pleasure.""".replace("\n", " ")
-
-  val b64Text = 
"""TWFuIGlzIGRpc3Rpbmd1aXNoZWQsIG5vdCBvbmx5IGJ5IGhpcyByZWFzb24sIGJ1dCBieSB0aGlz
-IHNpbmd1bGFyIHBhc3Npb24gZnJvbSBvdGhlciBhbmltYWxzLCB3aGljaCBpcyBhIGx1c3Qgb2Yg
-dGhlIG1pbmQsIHRoYXQgYnkgYSBwZXJzZXZlcmFuY2Ugb2YgZGVsaWdodCBpbiB0aGUgY29udGlu
-dWVkIGFuZCBpbmRlZmF0aWdhYmxlIGdlbmVyYXRpb24gb2Yga25vd2xlZGdlLCBleGNlZWRzIHRo
-ZSBzaG9ydCB2ZWhlbWVuY2Ugb2YgYW55IGNhcm5hbCBwbGVhc3VyZS4=""".replace("\n", 
"\r\n")
+  //
+  // This is from Don Quixote which is entirely public domain
+  //
 
 Review comment:
   Ok, well to simplify life I'll just change this text to the text of the 
Apache License!
   
   My only goal for this text is that it is readable, so that when debugging 
you can easily spot if it has been clobbered.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-19 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r182924768
 
 

 ##
 File path: 
daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/LayerTransformer.scala
 ##
 @@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthKind
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthUnits
+import org.apache.daffodil.processors.TermRuntimeData
+import org.apache.daffodil.processors.LayerLengthInBytesEv
+import org.apache.daffodil.processors.LayerTerminatorEv
+import org.apache.daffodil.processors.LayerCharsetEv
+import java.util.HashMap
+import org.apache.daffodil.processors.ParseOrUnparseState
+import org.apache.daffodil.util.NonAllocatingMap
+import org.apache.daffodil.io.DataInputStream
+import org.apache.daffodil.io.DataOutputStream
+import org.apache.daffodil.io.ByteBufferDataInputStream
+import org.apache.daffodil.io.DataInputStream.Mark
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.exceptions.Assert
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.util.Maybe.One
+import org.apache.daffodil.util.Maybe.Nope
+import org.apache.daffodil.io.FormatInfo
+import org.apache.daffodil.schema.annotation.props.gen.BitOrder
+
+/**
+ * Factory for a layer transformer.
+ *
+ * Responsible for digesting the various args, erroring if the wrong ones
+ * are specified, and ultimately constructing the LayerTransformer
+ * of the correct type with the parameters it needs.
+ */
+abstract class LayerTransformerFactory(nom: String)
+  extends Serializable {
+
+  val name = nom.toUpperCase()
+
+  def newInstance(maybeLayerCharsetEv: Maybe[LayerCharsetEv],
+maybeLayerLengthKind: Maybe[LayerLengthKind],
+maybeLayerLengthInBytesEv: Maybe[LayerLengthInBytesEv],
+maybeLayerLengthUnits: Maybe[LayerLengthUnits],
+maybeLayerTerminatorEv: Maybe[LayerTerminatorEv],
+trd: TermRuntimeData): LayerTransformer
+}
+
+/**
+ * Transformers have factories. This lets you find the transformer factory
+ * by the name obtained from dfdl:layerTransform.
+ */
+object LayerTransformerFactory {
+
+  private lazy val transformerMap = new NonAllocatingMap(new HashMap[String, 
LayerTransformerFactory])
+
+  def register(factory: LayerTransformerFactory): Unit = {
+transformerMap.put(factory.name, factory)
+  }
+  /**
+   * Given name, finds the factory for the transformer. SDE otherwise.
+   *
+   * The state is passed in order to provide diagnostic context if not found.
+   */
+  def find(name: String, state: ParseOrUnparseState): LayerTransformerFactory 
= {
+val maybeFactory = transformerMap.get(name)
+if (maybeFactory.isEmpty) {
+  val choices = transformerMap.keySet.mkString(", ")
+  state.SDE("The dfdl:layerTransform '%s' was not found. Available choices 
are: %s", name, choices)
+} else {
+  maybeFactory.get
+}
+  }
+
+  /**
+   * All transformers must be registered so they are available by name.
+   *
+   * It is possible to package a transformer in a separate jar also, but then
+   * someone has to register it by calling the register method.
+   *
+   * Transformers built into the primary Daffodil jars/packages should be
+   * registered here.
+   */
 
 Review comment:
   added https://issues.apache.org/jira/browse/DAFFODIL-1927 to record this 
feature request.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-19 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r182922848
 
 

 ##
 File path: 
daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/USASCII6BitPackedDecoder.scala
 ##
 @@ -26,15 +26,26 @@ import 
org.apache.daffodil.schema.annotation.props.gen.BitOrder
  * code unit.
  *
  */
-
-object USASCII6BitPackedCharset
-  extends NBitsWidth_BitsCharset("X-DFDL-US-ASCII-6-BIT-PACKED",
+private[charset] class USASCII6BitPacked(bitOrder: BitOrder, override val 
name: String)
+  extends NBitsWidth_BitsCharset(name,
 """@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_ !"#$%&'()*+,-./0123456789:;<=>?""",
 6, // width
-BitOrder.LeastSignificantBitFirst,
+bitOrder,
 0x1F) { // replacement charCode for encoding of unmapped chars.
   //
   // Note: formula for computing string above is
   // def decodeString = ((0 to 63).map { charCode => (if (charCode <= 31) 
charCode + 64 else charCode).toChar }).mkString
 }
 
+object USASCII6BitPackedLSBFirstCharset
+  extends USASCII6BitPacked(BitOrder.LeastSignificantBitFirst,
+"X-DFDL-US-ASCII-6-BIT-PACKED-LSB-FIRST")
+
+object USASCII6BitPackedCharset
+  extends USASCII6BitPacked(BitOrder.LeastSignificantBitFirst,
+"X-DFDL-US-ASCII-6-BIT-PACKED") // just an alias for the the "LSB-FIRST 
variant.
 
 Review comment:
   See https://issues.apache.org/jira/browse/DAFFODIL-1926
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-19 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r182921926
 
 

 ##
 File path: 
daffodil-io/src/main/scala/org/apache/daffodil/processors/charset/AIS_PAYLOAD_ARMORING.scala
 ##
 @@ -0,0 +1,28 @@
+package org.apache.daffodil.processors.charset
+
+import org.apache.daffodil.schema.annotation.props.gen.BitOrder
+
+/**
+ * Special purpose. This is not used for decoding anything.
+ * The encoder is used to convert strings using the characters
+ * allowed, into binary data using the AIS Payload Armoring
+ * described here:
+ *
+ * http://catb.org/gpsd/AIVDM.html#_aivdm_aivdo_payload_armoring
+ *
+ * To convert a string of length N, You will get 6N bits.
+ *
+ * The decoder can be used for unit testing, but the point of this class
+ * is to make the encoder available for use in un-doing the AIS Payload
+ * armoring.
+ *
+ * Always use with encodingErrorPolicy="error" since supplying a character
 
 Review comment:
   I'll improve the comment. We don't have the EEP='error' option, so wherever 
this AIS encoder gets called we'll need to do the checking ourselves. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-19 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r182921654
 
 

 ##
 File path: 
daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaOutputStreams.scala
 ##
 @@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.daffodil.io
+
+import java.nio.charset.Charset
+
 
 Review comment:
   How about the term "Boundary" instead of "terminator" to avoid confusion 
with the existing DFDL concept?
   So a layer can have a boundary mark, not a terminator. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-19 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r182921295
 
 

 ##
 File path: 
daffodil-io/src/main/scala/org/apache/daffodil/io/ByteBufferDataInputStream.scala
 ##
 @@ -91,7 +90,13 @@ object ByteBufferDataInputStream {
   case _ => {
 // copy the contents of the stream into an array of bytes
 val bos = new ByteArrayOutputStream
-IOUtils.copy(in, bos)
 
 Review comment:
   BTW I think this change was because i needed to breakpoint on the read or 
write when debugging.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-13 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r181540774
 
 

 ##
 File path: 
daffodil-core/src/main/scala/org/apache/daffodil/dsom/SequenceGroup.scala
 ##
 @@ -245,6 +248,31 @@ abstract class SequenceTermBase(
   maybeCheckBitOrderAndCharset)
   }
 
+  private val layeredSequenceAllowedProps = Set("ref", "layerTransform", 
"layerEncoding", "layerLengthKind", "layerLength", "layerLengthUnits", 
"layerTerminator")
+
+  final lazy val maybeLayerTransformerEv: Maybe[LayerTransformerEv] = {
+if (maybeLayerTransformEv.isEmpty) Maybe.Nope
 
 Review comment:
   You can't SDE because they could be in scope for use by layers elsewhere in 
the schema. 
   If they are expressed directly on this specific layered sequence, then we 
can SDW about them.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-13 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r181540700
 
 

 ##
 File path: 
daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/LayerTransformer.scala
 ##
 @@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthKind
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthUnits
+import org.apache.daffodil.processors.TermRuntimeData
+import org.apache.daffodil.processors.LayerLengthInBytesEv
+import org.apache.daffodil.processors.LayerTerminatorEv
+import org.apache.daffodil.processors.LayerCharsetEv
+import java.util.HashMap
+import org.apache.daffodil.processors.ParseOrUnparseState
+import org.apache.daffodil.util.NonAllocatingMap
+import org.apache.daffodil.io.DataInputStream
+import org.apache.daffodil.io.DataOutputStream
+import org.apache.daffodil.io.ByteBufferDataInputStream
+import org.apache.daffodil.io.DataInputStream.Mark
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.exceptions.Assert
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.util.Maybe.One
+import org.apache.daffodil.util.Maybe.Nope
+import org.apache.daffodil.io.FormatInfo
+import org.apache.daffodil.schema.annotation.props.gen.BitOrder
+
+/**
+ * Factory for a layer transformer.
+ *
+ * Responsible for digesting the various args, erroring if the wrong ones
+ * are specified, and ultimately constructing the LayerTransformer
+ * of the correct type with the parameters it needs.
+ */
+abstract class LayerTransformerFactory(nom: String)
+  extends Serializable {
+
+  val name = nom.toUpperCase()
+
+  def newInstance(maybeLayerCharsetEv: Maybe[LayerCharsetEv],
+maybeLayerLengthKind: Maybe[LayerLengthKind],
+maybeLayerLengthInBytesEv: Maybe[LayerLengthInBytesEv],
+maybeLayerLengthUnits: Maybe[LayerLengthUnits],
+maybeLayerTerminatorEv: Maybe[LayerTerminatorEv],
+trd: TermRuntimeData): LayerTransformer
+}
+
+/**
+ * Transformers have factories. This lets you find the transformer factory
+ * by the name obtained from dfdl:layerTransform.
+ */
+object LayerTransformerFactory {
+
+  private lazy val transformerMap = new NonAllocatingMap(new HashMap[String, 
LayerTransformerFactory])
+
+  def register(factory: LayerTransformerFactory): Unit = {
+transformerMap.put(factory.name, factory)
+  }
+  /**
+   * Given name, finds the factory for the transformer. SDE otherwise.
+   *
+   * The state is passed in order to provide diagnostic context if not found.
+   */
+  def find(name: String, state: ParseOrUnparseState): LayerTransformerFactory 
= {
+val maybeFactory = transformerMap.get(name)
+if (maybeFactory.isEmpty) {
+  val choices = transformerMap.keySet.mkString(", ")
+  state.SDE("The dfdl:layerTransform '%s' was not found. Available choices 
are: %s", name, choices)
+} else {
+  maybeFactory.get
+}
+  }
+
+  /**
+   * All transformers must be registered so they are available by name.
+   *
+   * It is possible to package a transformer in a separate jar also, but then
+   * someone has to register it by calling the register method.
+   *
+   * Transformers built into the primary Daffodil jars/packages should be
+   * registered here.
+   */
 
 Review comment:
   I'd like to be able to add transformers on the fly. Not high prio for now, 
but I am pretty sure it will come up in the future. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-13 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r181540674
 
 

 ##
 File path: 
daffodil-io/src/test/scala/org/apache/daffodil/layers/TestBase64.scala
 ##
 @@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import junit.framework.Assert._
+import org.junit.Test
+import org.apache.daffodil.Implicits._
+import org.apache.commons.io.IOUtils
+import org.apache.daffodil.exceptions.Assert
+
+/**
+ * There are 3 variations of base64 we care about.
+ *
+ * Base64_RFC2045 (MIME)
+ * Base64_RFC4648
+ * Base64_RFC4648_URLSAFE
+ *
+ * When decoding from Base64, the CRLF vs. LF doesn't matter. However when
+ * encoding, the RFC specify that CRLF are inserted.
+ */
+class TestBase64 {
+  Assert.usage(scala.util.Properties.isJavaAtLeast("1.8"))
+
+  val text = """Man is distinguished, not only by his reason, but by this 
singular passion from
+other animals, which is a lust of the mind, that by a perseverance of delight
+in the continued and indefatigable generation of knowledge, exceeds the short
+vehemence of any carnal pleasure.""".replace("\n", " ")
 
 Review comment:
   OK, I'll change it to Ipsum lorem or some other gibberish of my own making.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-13 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r181540295
 
 

 ##
 File path: 
daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaIOStreams.scala
 ##
 @@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.io
+
+import org.apache.daffodil.exceptions.Assert
+import java.util.regex.Pattern
+import java.util.Scanner
+import java.nio.charset.StandardCharsets
+import java.nio.charset.Charset
+import java.io.InputStream
+import java.io.FilterInputStream
+import java.io.InputStreamReader
+
+/**
+ * This class can be used with any InputStream to restrict what is
+ * read from it to N bytes.
+ *
+ * This can be used to forcibly stop consumption of data from a stream at
+ * a length obtained explicitly.
+ *
+ * Thread safety: This is inherently stateful - so not thread safe to use
+ * this object from more than one thread.
+ */
+class ExplicitLengthLimitingStream(in: InputStream, limit: Int)
+  extends FilterInputStream(in) {
+
+  private var numRemaining = limit
+
+  override def read(buf: Array[Byte], off: Int, len: Int) = {
+Assert.invariant(numRemaining >= 0)
+if (numRemaining == 0) -1
+else if (len == 0) 0
+else {
+  val requestSize = math.min(numRemaining, len)
+  val actualSize = in.read(buf, off, requestSize)
+  if (actualSize == -1)
+numRemaining = 0
+  else
+numRemaining -= actualSize
+  actualSize
+}
+  }
+
+  private val readBuf = new Array[Byte](1)
+
+  override def read(): Int = {
+readBuf(0) = 0
+val n = read(readBuf, 0, 1)
+if (n == -1)
+  -1
+else {
+  Assert.invariant(n == 1)
+  val i = readBuf(0).toInt
+  val b = i & 0xFF
+  b
+}
+  }
+}
+
+/**
+ * This class can be used with any InputStream to restrict what is
+ * read from it to stop before a terminator string.
+ *
+ * The terminator string is exactly that, a string of characters. Not a
+ * regex, nor anything involving DFDL Character Entities or Character Class
+ * Entities. (No %WSP; no %NL; )
+ *
+ * This can be used to forcibly stop consumption of data from a stream at
+ * a length obtained from a delimiter.
+ *
+ * The terminator string is consumed from the underlying stream (if found), and
+ * the underlying stream is left positioned at the byte after the terminator
+ * string.
+ *
+ * Thread safety: This is inherently stateful - so not thread safe to use
+ * this object from more than one thread.
+ */
+class DelimitedLengthLimitingStream(inWithMarkSupported: InputStream, 
terminator: String,
+  charset: Charset = StandardCharsets.ISO_8859_1,
+  maxChunkSize: Int = 1024 * 32) // 32K chunk size by default. Units are chars.
+  extends FilterInputStream(inWithMarkSupported) {
+  Assert.usage(maxChunkSize >= 1)
+
+  private val charsetDecoder = charset.newDecoder()
+  private val rdr = new InputStreamReader(in, charsetDecoder)
+  private val maxBytesPerChar = charset.newEncoder().maxBytesPerChar()
+
+  /**
+   * We will create a new scanner for the rdr each read call, because
+   * scanners are stateful.
+   */
+  private var scanner: Scanner = _ // TODO: Performance... how to avoid this 
allocation overhead? Implement our own scanner?
+
+  /**
+   * This pattern uses unsafe ".*" but since it is always used along with
+   * the scanner methods that take a horizon limit, it is safe.
+   *
+   * Pattern matches the shortest string of anything followed by the 
terminator.
+   * The part preceding the terminator goes into capture group 1.
+   *
+   * After we match this regex with the scanner,
+   * the input stream should end up positioned immediately before the 
terminator.
+   * The terminator is not part of the match, because it is in a non-capturing
+   * lookahead group.
+   *
+   * However, for assertion checking, a second capture group is defined
+   * inside the lookahead, and that should capture the terminator.
+   */
+  private val quotedTerminator = Pattern.quote(terminator) // in case pattern 
has non-regex-safe characters in it
 
 Review comment:
   No specific reason. I was 

[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-13 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r181540210
 
 

 ##
 File path: 
daffodil-io/src/main/scala/org/apache/daffodil/io/LimitingJavaIOStreams.scala
 ##
 @@ -0,0 +1,281 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.io
+
+import org.apache.daffodil.exceptions.Assert
+import java.util.regex.Pattern
+import java.util.Scanner
+import java.nio.charset.StandardCharsets
+import java.nio.charset.Charset
+import java.io.InputStream
+import java.io.FilterInputStream
+import java.io.InputStreamReader
+
+/**
+ * This class can be used with any InputStream to restrict what is
+ * read from it to N bytes.
+ *
+ * This can be used to forcibly stop consumption of data from a stream at
+ * a length obtained explicitly.
+ *
+ * Thread safety: This is inherently stateful - so not thread safe to use
+ * this object from more than one thread.
+ */
+class ExplicitLengthLimitingStream(in: InputStream, limit: Int)
+  extends FilterInputStream(in) {
+
+  private var numRemaining = limit
+
+  override def read(buf: Array[Byte], off: Int, len: Int) = {
+Assert.invariant(numRemaining >= 0)
+if (numRemaining == 0) -1
+else if (len == 0) 0
+else {
+  val requestSize = math.min(numRemaining, len)
+  val actualSize = in.read(buf, off, requestSize)
+  if (actualSize == -1)
+numRemaining = 0
+  else
+numRemaining -= actualSize
+  actualSize
+}
+  }
+
+  private val readBuf = new Array[Byte](1)
+
+  override def read(): Int = {
+readBuf(0) = 0
+val n = read(readBuf, 0, 1)
+if (n == -1)
+  -1
+else {
+  Assert.invariant(n == 1)
+  val i = readBuf(0).toInt
+  val b = i & 0xFF
+  b
+}
+  }
+}
+
+/**
+ * This class can be used with any InputStream to restrict what is
+ * read from it to stop before a terminator string.
+ *
+ * The terminator string is exactly that, a string of characters. Not a
+ * regex, nor anything involving DFDL Character Entities or Character Class
+ * Entities. (No %WSP; no %NL; )
+ *
+ * This can be used to forcibly stop consumption of data from a stream at
+ * a length obtained from a delimiter.
+ *
+ * The terminator string is consumed from the underlying stream (if found), and
+ * the underlying stream is left positioned at the byte after the terminator
+ * string.
+ *
+ * Thread safety: This is inherently stateful - so not thread safe to use
+ * this object from more than one thread.
+ */
+class DelimitedLengthLimitingStream(inWithMarkSupported: InputStream, 
terminator: String,
+  charset: Charset = StandardCharsets.ISO_8859_1,
+  maxChunkSize: Int = 1024 * 32) // 32K chunk size by default. Units are chars.
+  extends FilterInputStream(inWithMarkSupported) {
 
 Review comment:
   I think each layer DataInputStream, or Java.io.InputStream has to support 
mark/reset. When we wrap a JavaIOInputStream around a DataInputStream, we 
implement the mark/reset in terms of what DataInputStream supports. 
   
   The need for mark/reset here is for a very knarly painful thing about the 
way I'm using Java Scanner class to search for a regex. Because this operats on 
a reader, and both it and the reader layer read ahead a bit, we have to play 
tricks to get the underlying input stream to be precisely positioned 
immediately after the last byte that was "required" to decode the layer. 
   
   More in next commit. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-13 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r181539996
 
 

 ##
 File path: 
daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/LayerTransformer.scala
 ##
 @@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthKind
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthUnits
+import org.apache.daffodil.processors.TermRuntimeData
+import org.apache.daffodil.processors.LayerLengthInBytesEv
+import org.apache.daffodil.processors.LayerTerminatorEv
+import org.apache.daffodil.processors.LayerCharsetEv
+import java.util.HashMap
+import org.apache.daffodil.processors.ParseOrUnparseState
+import org.apache.daffodil.util.NonAllocatingMap
+import org.apache.daffodil.io.DataInputStream
+import org.apache.daffodil.io.DataOutputStream
+import org.apache.daffodil.io.ByteBufferDataInputStream
+import org.apache.daffodil.io.DataInputStream.Mark
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.exceptions.Assert
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.util.Maybe.One
+import org.apache.daffodil.util.Maybe.Nope
+import org.apache.daffodil.io.FormatInfo
+import org.apache.daffodil.schema.annotation.props.gen.BitOrder
+
+/**
+ * Factory for a layer transformer.
+ *
+ * Responsible for digesting the various args, erroring if the wrong ones
+ * are specified, and ultimately constructing the LayerTransformer
+ * of the correct type with the parameters it needs.
+ */
+abstract class LayerTransformerFactory(nom: String)
+  extends Serializable {
+
+  val name = nom.toUpperCase()
+
+  def newInstance(maybeLayerCharsetEv: Maybe[LayerCharsetEv],
+maybeLayerLengthKind: Maybe[LayerLengthKind],
+maybeLayerLengthInBytesEv: Maybe[LayerLengthInBytesEv],
+maybeLayerLengthUnits: Maybe[LayerLengthUnits],
+maybeLayerTerminatorEv: Maybe[LayerTerminatorEv],
+trd: TermRuntimeData): LayerTransformer
+}
+
+/**
+ * Transformers have factories. This lets you find the transformer factory
+ * by the name obtained from dfdl:layerTransform.
+ */
+object LayerTransformerFactory {
+
+  private lazy val transformerMap = new NonAllocatingMap(new HashMap[String, 
LayerTransformerFactory])
+
+  def register(factory: LayerTransformerFactory): Unit = {
+transformerMap.put(factory.name, factory)
+  }
+  /**
+   * Given name, finds the factory for the transformer. SDE otherwise.
+   *
+   * The state is passed in order to provide diagnostic context if not found.
+   */
+  def find(name: String, state: ParseOrUnparseState): LayerTransformerFactory 
= {
+val maybeFactory = transformerMap.get(name)
+if (maybeFactory.isEmpty) {
+  val choices = transformerMap.keySet.mkString(", ")
+  state.SDE("The dfdl:layerTransform '%s' was not found. Available choices 
are: %s", name, choices)
+} else {
+  maybeFactory.get
+}
+  }
+
+  /**
+   * All transformers must be registered so they are available by name.
+   *
+   * It is possible to package a transformer in a separate jar also, but then
+   * someone has to register it by calling the register method.
+   *
+   * Transformers built into the primary Daffodil jars/packages should be
+   * registered here.
+   */
+  register(Base64MIMETransformerFactory)
+}
+
+/**
+ * Shared functionality of all LayerTransformers.
+ */
+abstract class LayerTransformer()
+  extends Serializable {
+
+  protected def wrapLayerDecoder(jis: java.io.InputStream): java.io.InputStream
+
+  def wrapJavaInputStream(s: DataInputStream, fInfo: FormatInfo): 
java.io.InputStream = {
+new JavaIOInputStream(s, fInfo)
+  }
+
+  /**
+   *  Override these if we ever have transformers that don't have these
+   *  requirements.
+   */
+  val mandatoryLayerAlignmentInBits: Int = 8
+  val mandatoryLengthUnit: LayerLengthUnits = LayerLengthUnits.Bytes
+
+  def addLayer(s: DataInputStream, fInfo: FormatInfo): DataInputStream = {
+val jis = wrapJavaInputStream(s, fInfo)
+val decodedInputStream = 

[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-11 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r180913080
 
 

 ##
 File path: 
daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/LayerTransformer.scala
 ##
 @@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.layers
+
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthKind
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthUnits
+import org.apache.daffodil.processors.TermRuntimeData
+import org.apache.daffodil.processors.LayerLengthInBytesEv
+import org.apache.daffodil.processors.LayerTerminatorEv
+import org.apache.daffodil.processors.LayerCharsetEv
+import java.util.HashMap
+import org.apache.daffodil.processors.ParseOrUnparseState
+import org.apache.daffodil.util.NonAllocatingMap
+import org.apache.daffodil.io.DataInputStream
+import org.apache.daffodil.io.DataOutputStream
+import org.apache.daffodil.io.ByteBufferDataInputStream
+import org.apache.daffodil.io.DataInputStream.Mark
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.exceptions.Assert
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.util.Maybe.One
+import org.apache.daffodil.util.Maybe.Nope
+import org.apache.daffodil.io.FormatInfo
+import org.apache.daffodil.schema.annotation.props.gen.BitOrder
+
+/**
+ * Factory for a layer transformer.
+ *
+ * Responsible for digesting the various args, erroring if the wrong ones
+ * are specified, and ultimately constructing the LayerTransformer
+ * of the correct type with the parameters it needs.
+ */
+abstract class LayerTransformerFactory(nom: String)
+  extends Serializable {
+
+  val name = nom.toUpperCase()
+
+  def newInstance(maybeLayerCharsetEv: Maybe[LayerCharsetEv],
+maybeLayerLengthKind: Maybe[LayerLengthKind],
+maybeLayerLengthInBytesEv: Maybe[LayerLengthInBytesEv],
+maybeLayerLengthUnits: Maybe[LayerLengthUnits],
+maybeLayerTerminatorEv: Maybe[LayerTerminatorEv],
+trd: TermRuntimeData): LayerTransformer
+}
+
+/**
+ * Transformers have factories. This lets you find the transformer factory
+ * by the name obtained from dfdl:layerTransform.
+ */
+object LayerTransformerFactory {
+
+  private lazy val transformerMap = new NonAllocatingMap(new HashMap[String, 
LayerTransformerFactory])
+
+  def register(factory: LayerTransformerFactory): Unit = {
+transformerMap.put(factory.name, factory)
+  }
+  /**
+   * Given name, finds the factory for the transformer. SDE otherwise.
+   *
+   * The state is passed in order to provide diagnostic context if not found.
+   */
+  def find(name: String, state: ParseOrUnparseState): LayerTransformerFactory 
= {
+val maybeFactory = transformerMap.get(name)
+if (maybeFactory.isEmpty) {
+  val choices = transformerMap.keySet.mkString(", ")
+  state.SDE("The dfdl:layerTransform '%s' was not found. Available choices 
are: %s", name, choices)
+} else {
+  maybeFactory.get
+}
+  }
+
+  /**
+   * All transformers must be registered so they are available by name.
+   *
+   * It is possible to package a transformer in a separate jar also, but then
+   * someone has to register it by calling the register method.
+   *
+   * Transformers built into the primary Daffodil jars/packages should be
+   * registered here.
+   */
+  register(Base64MIMETransformerFactory)
+}
+
+/**
+ * Shared functionality of all LayerTransformers.
+ */
+abstract class LayerTransformer()
+  extends Serializable {
+
+  protected def wrapLayerDecoder(jis: java.io.InputStream): java.io.InputStream
+
+  def wrapJavaInputStream(s: DataInputStream, fInfo: FormatInfo): 
java.io.InputStream = {
+new JavaIOInputStream(s, fInfo)
 
 Review comment:
   We need to additionally wrap an Explicit or Delimited Limiting Stream around 
this JavaIOInputStream. Otherwise the decoding will not know when to stop. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 

[GitHub] mbeckerle commented on a change in pull request #61: Preliminary Review - Base64 layering runs first parsing unit test.

2018-04-11 Thread GitBox
mbeckerle commented on a change in pull request #61: Preliminary Review - 
Base64 layering runs first parsing unit test.
URL: https://github.com/apache/incubator-daffodil/pull/61#discussion_r180909573
 
 

 ##
 File path: 
daffodil-runtime1/src/main/scala/org/apache/daffodil/layers/Base64Transformer.scala
 ##
 @@ -0,0 +1,43 @@
+package org.apache.daffodil.layers
+
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthKind
+import org.apache.daffodil.schema.annotation.props.gen.LayerLengthUnits
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.processors.TermRuntimeData
+import org.apache.daffodil.processors.LayerLengthInBytesEv
+import org.apache.daffodil.processors.LayerTerminatorEv
+import org.apache.daffodil.processors.LayerCharsetEv
+
+class Base64MIMETransformer(layerCharsetEv: LayerCharsetEv, layerTerminatorEv: 
LayerTerminatorEv)
+  extends LayerTransformer() {
+
+  override def wrapLayerDecoder(jis: java.io.InputStream) = {
+val b64 = java.util.Base64.getMimeDecoder().wrap(jis)
+b64
+  }
+
+}
+
+object Base64MIMETransformerFactory
 
 Review comment:
   Add scaladoc: newInstance method digests the various layering related 
arguments to decide how to configure and which layer transformer to create, 
then constructs it appropriately.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services