tuxji commented on a change in pull request #532:
URL: https://github.com/apache/daffodil/pull/532#discussion_r616132788
##########
File path: .github/workflows/main.yml
##########
@@ -25,30 +25,64 @@ jobs:
matrix:
java_version: [ 8, 11, 16 ]
scala_version: [ 2.12.11 ]
- os: [ 'ubuntu-latest', 'windows-latest' ]
- env:
- SBT: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }} coverage
- SBTNOCOV: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }}
+ os: [ ubuntu-20.04, windows-2019 ]
+ include:
+ - os: ubuntu-20.04
+ shell: bash
+ - os: windows-2019
+ shell: msys2 {0}
runs-on: ${{ matrix.os }}
defaults:
run:
- shell: bash
-
+ shell: ${{ matrix.shell }}
+ env:
+ SBT: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }} coverage
+ SBTNOCOV: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }}
steps:
############################################################
# Setup
############################################################
- - name: Checkout Repository
- uses: actions/[email protected]
+ - name: Install Dependencies (Linux)
+ if: runner.os == 'Linux'
+ run: sudo apt-get install -y libmxml-dev
Review comment:
Yes, we only use mini-xml. This is not libxml-dev; this is libmxml-dev
(note the 'm' character between 'lib' and 'xml').
##########
File path: .github/workflows/main.yml
##########
@@ -25,30 +25,64 @@ jobs:
matrix:
java_version: [ 8, 11, 16 ]
scala_version: [ 2.12.11 ]
- os: [ 'ubuntu-latest', 'windows-latest' ]
- env:
- SBT: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }} coverage
- SBTNOCOV: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }}
+ os: [ ubuntu-20.04, windows-2019 ]
+ include:
+ - os: ubuntu-20.04
+ shell: bash
+ - os: windows-2019
+ shell: msys2 {0}
runs-on: ${{ matrix.os }}
defaults:
run:
- shell: bash
-
+ shell: ${{ matrix.shell }}
+ env:
+ SBT: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }} coverage
+ SBTNOCOV: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }}
steps:
############################################################
# Setup
############################################################
- - name: Checkout Repository
- uses: actions/[email protected]
+ - name: Install Dependencies (Linux)
+ if: runner.os == 'Linux'
+ run: sudo apt-get install -y libmxml-dev
+
+ - name: Install Dependencies (Windows)
+ if: runner.os == 'Windows'
+ uses: msys2/setup-msys2@v2
+ with:
+ install: gcc libargp-devel make pkgconf
+ path-type: inherit
- - name: Install Java
- uses: actions/setup-java@v1
+ - name: Check out mxml source (Windows)
+ if: runner.os == 'Windows'
+ uses: actions/[email protected]
+ with:
+ repository: michaelrsweet/mxml
+ ref: v3.2
+ path: mxml
+
+ - name: Install mxml library (Windows)
+ if: runner.os == 'Windows'
+ run: |
+ cd mxml
+ ./configure --prefix=/usr --disable-shared --disable-threads
+ make
+ make install
+ # Workaround for sbt hanging problem
+ echo "COURSIER_CACHE=$temp" >> $GITHUB_ENV
+ echo "COURSIER_CONFIG_DIR=$temp" >> $GITHUB_ENV
Review comment:
Yes, it seems like a good idea to cache some build files that coursier
and sbt normally cache on a development machine anyway. However, I see that
coursier/cache-action will define only the COURSIER_CACHE environment variable
for us; it won't define the COURSIER_CONFIG_DIR environment variable which is
also necessary to stop coursier from calling dev.dirs'
ProjectDirectories.from(null, null, "Coursier") method, which is what causes
sbt to hang on Windows. Therefore, we can't remove these "echo >> $GITHUB_ENV"
lines until the hanging problem is fixed in dev.dirs' directories library,
coursier picks up the new directories version, and sbt picks up the new
coursier version, which could be a long time from now.
##########
File path: BUILD.md
##########
@@ -0,0 +1,114 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+# Build Requirements
+
+Daffodil's build requirements include:
+
+* JDK 8 or higher
+* SBT 0.13.8 or higher
+* C compiler C99 or higher
+* Mini-XML Version 3.2 or higher
+
+You will need the Java Software Development Kit ([JDK]) and the Scala
+Build Tool ([SBT]) to build Daffodil, run all tests, create packages,
+and more. You can install the latest [Java 11 LTS][JDK] version and
+the latest [SBT] version following their websites' instructions or
+install them using your operating system's package manager.
+
+Since Daffodil now has a C backend as well as a Scala backend, you
+will need a C compiler supporting the [C99] standard or later, the
+[Mini-XML] library, and possibly the GNU [argp] library if your
+system's C library doesn't include it already. You can install either
+[gcc] or [clang] using your operating system's package manager. If
+you can't install the [Mini-XML] library using your operating system's
+package manager, you'll have to build it from source. We'll tell you
+how to do that on Windows but the commands should work on other
+operating systems too.
+
+You can set your environment variables "CC" and "AR" to the correct
+commands (or set them to `true` to disable C compilation altogether)
+if you don't want `sbt compile` to call your C compiler with `cc` and
+`ar` as the default commands.
+
+## Fedora 33
+
+You can use the `dnf` package manager to install most of the build
+requirements needed to build Daffodil:
+
+ sudo dnf install gcc git java-11-openjdk-devel make mxml-devel pkgconf
+
+However, Fedora has no sbt package in its default repositories.
+You'll have to install the latest [SBT] version following its
+website's instructions.
+
+Now you can build Daffodil from source and the sbt and daffodil
+commands you type will be able to call the C compiler.
+
+## Ubuntu 20.04
+
+You can use the `apt` package manager to install most of the build
+requirements needed to build Daffodil:
+
+ sudo apt install build-essential default-jdk git libmxml-dev
+
+However, Ubuntu has no sbt package in its default repositories.
+You'll have to install the latest [SBT] version following its
+website's instructions.
+
+Now you can build Daffodil from source and the sbt and daffodil
+commands you type will be able to call the C compiler.
+
+## Windows 10
+
+Install the latest [Java 11 LTS][JDK] version and the latest [SBT]
+version following their websites' instructions.
+
+Install [MSYS2] following its website's instructions and open a new
+"MSYS2 MSYS" window. We'll need its collection of free programs and
+libraries.
+
+Install [gcc] and [libargp][argp] using MSYS2's `pacman` package
+manager:
+
+ pacman -S gcc git libargp-devel make pkgconf
+
+However, MSYS2 has no [libmxml-devel][Mini-XML] package so you'll have
Review comment:
Note the 'm' character; libmxml-devel is indeed the Mini-XML library (or
would be if someone added that package to MSYS2).
##########
File path:
daffodil-runtime1/src/main/scala/org/apache/daffodil/api/DFDLParserUnparser.scala
##########
@@ -147,11 +149,41 @@ object DFDL {
@deprecated("Use arguments to Compiler.compileSource or compileFile.",
"2.6.0")
def setDistinguishedRootNode(name: String, namespace: String = null): Unit
+ /**
+ * Returns a [[DataProcessor]] to process data matching a compiled XPath
expression
+ * @param xpath XPath expression in DFDL schema that data should match
(you can use only "/" at this time)
+ */
def onPath(xpath: String): DataProcessor
+
+ /**
+ * Returns a [[CodeGenerator]] to generate code from a DFDL schema to
parse or unparse data
+ * @param language source language for generated code (you can use only
"c" at this time)
+ */
+ def forLanguage(language: String): CodeGenerator
}
- trait DataProcessor extends WithDiagnostics {
+ /**
+ * Source code generation and compilation is performed with a
language-specific [[CodeGenerator]],
+ * which must be interrogated for diagnostics to see if each call was
successful or not.
+ */
+ trait CodeGenerator extends WithDiagnostics {
+ /**
+ * Generates language-specific code from a DFDL schema to parse or unparse
data
+ * @param rootNS one of the top-level elements of the DFDL schema (if not
supplied, uses first top-level element)
+ * @param outputDir output directory in which to generate code
+ * @return path of output directory containing generated code
+ */
+ def generateCode(rootNS: Option[RefQName], outputDir: String): os.Path
+
+ /**
+ * Compiles the generated code so it can be used by a TDML test or
something else
+ * @param outputDir path of output directory containing generated code
+ * @return path of executable built from generated code
+ */
+ def compileCode(outputDir: os.Path): os.Path
Review comment:
Yes, you need to pass the generateCode method's return value to the
compileCode method as its input parameter. I agree I gave the input parameter
a confusing name. In fact, when I implemented both methods in
CodeGenerator.scala, I used the name "codeDir" for both generateCode's return
value and compileCode's input parameter. I've updated trait CodeGenerator to
use the name "codeDir" in both places as well.
Although we could pass compileCode a path to the executable, I think it
would be unnecessary for two reasons. The user has only two entry points from
Daffodil's CLI ("generate" and "test"). The "generate" entry point calls
generateCode but doesn't call compileCode because our use case requires only
the libruntime files, so there's no reason to build an executable that we don't
need. Just in case a user really does want to build the executable, the
codeDir has a Makefile so all the user has to type is "cd c && make".
Likewise, the user has no need to specify where the executable is built when
running "daffodil test". The TDML Runner creates a temporary directory, calls
generateCode and compileCode, runs the executable, and deletes the temporary
directory without any user involvement needed. All compileCode needs is to
pick the right name for the exe and return it as its return value:
```scala
val exe = if (isWindows) codeDir/"daffodil.exe" else codeDir/"daffodil"
```
##########
File path: daffodil-runtime2/src/main/resources/.clang-format
##########
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+AlignConsecutiveDeclarations: true
+AllowShortFunctionsOnASingleLine: None
+AllowShortIfStatementsOnASingleLine: true
+AlwaysBreakAfterReturnType: TopLevelDefinitions
+BasedOnStyle: llvm
+BreakBeforeBraces: Allman
+ColumnLimit: 110
+IndentWidth: 4
+KeepEmptyLinesAtTheStartOfBlocks: false
Review comment:
All the C files are below this resources directory so this configuration
file doesn't need to be placed any higher. I don't want to clutter the repo
root unnecessarily so let me know if you'd rather have me put the clang-format
file in the repo root.
##########
File path: daffodil-runtime2/src/main/resources/c/libcli/daffodil_argp.c
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "daffodil_argp.h"
+#include <argp.h> // for argp_state, argp_error, error_t, argp_parse,
ARGP_ERR_UNKNOWN, ARGP_IN_ORDER, ARGP_KEY_ARG, argp, argp_option, ARGP_KEY_END
Review comment:
Do you still mind the GNU license after I answered your question about
how much of a dependency Daffodil has on argp (needed only when test-building C
files and running tests)? What makes argp as nice as any other arg library is
that it supports both long options (getopt doesn't) and subcommands
(getopt_long makes it hard with some problems) and it's widely available thanks
to its inclusion in the glibc library and prebuilt packages on other systems.
I did find a possible alternative just now (argtable3, MIT license,
[multisyntax example
here](https://github.com/argtable/argtable3/blob/master/examples/multisyntax.c)),
but argtable3 would have to be included in Daffodil since it's a fairly new
library and as not widely available as argp.
##########
File path: daffodil-runtime2/src/main/resources/c/libcli/xml_writer.c
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "xml_writer.h"
+#include <assert.h> // for assert
+#include <mxml.h> // for mxmlNewOpaquef, mxml_node_t, mxmlElementSetAttr,
mxmlGetOpaque, mxmlNewElement, mxmlDelete, mxmlGetElement, mxmlNewXML,
mxmlSaveFile, MXML_NO_CALLBACK
+#include <stdbool.h> // for bool
+#include <stdint.h> // for int16_t, int32_t, int64_t, int8_t, uint16_t,
uint32_t, uint64_t, uint8_t
+#include <stdio.h> // for NULL
+#include <string.h> // for strcmp
+#include "errors.h" // for Error, ERR_XML_DECL, ERR_XML_ELEMENT,
ERR_XML_WRITE, LIMIT_XML_NESTING, Error::(anonymous)
+#include "stack.h" // for stack_is_empty, stack_pop, stack_push, stack_top,
stack_init
+
+// Push new XML document on stack (note the stack is stored in a
+// static array which could overflow and stop the program; it also
+// means none of those functions are thread-safe)
+
+static const Error *
+xmlStartDocument(XMLWriter *writer)
+{
+ static mxml_node_t *array[LIMIT_XML_NESTING];
+ stack_init(&writer->stack, array, LIMIT_XML_NESTING);
+
+ mxml_node_t *xml = mxmlNewXML("1.0");
+ if (xml)
+ {
+ stack_push(&writer->stack, xml);
+ return NULL;
+ }
+ else
+ {
+ static Error error = {ERR_XML_DECL, {NULL}};
+ return &error;
+ }
+}
+
+// Pop completed XML document off stack and write it to stream (note
+// stack underflow will stop program)
+
+static const Error *
+xmlEndDocument(XMLWriter *writer)
+{
+ mxml_node_t *xml = stack_pop(&writer->stack);
+ assert(stack_is_empty(&writer->stack));
+
+ int status = mxmlSaveFile(xml, writer->stream, MXML_NO_CALLBACK);
+ if (status < 0)
+ {
+ static Error error = {ERR_XML_WRITE, {NULL}};
+ return &error;
+ }
+ mxmlDelete(xml);
+ return NULL;
+}
+
+// Push new complex element on stack (note stack overflow will stop
+// program)
+
+static const Error *
+xmlStartComplex(XMLWriter *writer, const InfosetBase *base)
+{
+ mxml_node_t *parent = stack_top(&writer->stack);
+ const char * name = get_erd_name(base->erd);
+ const char * xmlns = get_erd_xmlns(base->erd);
+ mxml_node_t *complex = mxmlNewElement(parent, name);
+ if (xmlns)
+ {
+ const char *ns = get_erd_ns(base->erd);
+ mxmlElementSetAttr(complex, xmlns, ns);
+ }
+ stack_push(&writer->stack, complex);
+ return NULL;
+}
+
+// Pop completed complex element off stack (note stack underflow will
+// stop program)
+
+static const Error *
+xmlEndComplex(XMLWriter *writer, const InfosetBase *base)
+{
+ mxml_node_t *complex = stack_pop(&writer->stack);
+
+ const char *name_from_xml = mxmlGetElement(complex);
+ const char *name_from_erd = get_erd_name(base->erd);
+ assert(strcmp(name_from_xml, name_from_erd) == 0);
+
+ return NULL;
+}
+
+// Fix a real number to conform to xsd:float syntax if needed
+
+static void
+fixNumberIfNeeded(const char *text)
+{
+ if (text[0] == 'N' && text[1] == 'A')
+ {
+ // xsd:float requires NaN to be capitalized correctly
+ char *modifyInPlace = (char *)text;
+ modifyInPlace[1] = 'a';
+ }
+ // These are not required by xsd:float, only to match runtime1 better
+ // - Strip + from <f>E+<e> to get <f>E<e>
+ // - Add .0 to 1 to get 1.0
+ // It would be better to compare floats as numbers, not strings, though
Review comment:
My comment is referring to possible code that could've been, but wasn't,
written too because 1) there's still a third difference (our numbers preserve
more significant digits) and 2) comparing numbers numerically, not
alphabetically, would make harmless textual differences irrelevant anyway.
I've updated the comment to say so more clearly.
##########
File path: daffodil-runtime2/src/main/resources/c/libruntime/errors.c
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "errors.h"
+#include <assert.h> // for assert
+#include <error.h> // for error
+#include <inttypes.h> // for PRId64
+#include <stdbool.h> // for bool, false, true
+#include <stdio.h> // for NULL, feof, ferror, FILE, size_t
+#include <stdlib.h> // for EXIT_FAILURE
+
+// error_message - return an internationalized error message
+
+static const char *
+error_message(enum ErrorCode code)
+{
+ switch (code)
+ {
+ case ERR_CHOICE_KEY:
+ return "no match between choice dispatch key %" PRId64 " and any
branch key";
+ case ERR_FILE_CLOSE:
+ return "error closing file";
+ case ERR_FILE_FLUSH:
+ return "error flushing stream to file";
+ case ERR_FILE_OPEN:
+ return "error opening file '%s'";
+ case ERR_FIXED_VALUE:
+ return "value of element '%s' does not match value of its "
+ "'fixed' attribute";
+ case ERR_INFOSET_READ:
+ return "cannot read infoset type '%s'";
+ case ERR_INFOSET_WRITE:
+ return "cannot write infoset type '%s'";
+ case ERR_PARSE_BOOL:
+ return "error parsing binary value %" PRId64 " as either true or
false";
+ case ERR_STACK_EMPTY:
+ return "stack empty, stopping program";
+ case ERR_STACK_OVERFLOW:
+ return "stack overflow, stopping program";
+ case ERR_STACK_UNDERFLOW:
+ return "stack underflow, stopping program";
+ case ERR_STREAM_EOF:
+ return "EOF in stream, stopping program";
+ case ERR_STREAM_ERROR:
+ return "error in stream, stopping program";
+ case ERR_STRTOBOOL:
+ return "error converting XML data '%s' to boolean";
+ case ERR_STRTOD_ERRNO:
+ return "error converting XML data '%s' to number";
+ case ERR_STRTOI_ERRNO:
+ return "error converting XML data '%s' to integer";
+ case ERR_STRTONUM_EMPTY:
+ return "found no number in XML data '%s'";
+ case ERR_STRTONUM_NOT:
+ return "found non-number characters in XML data '%s'";
+ case ERR_STRTONUM_RANGE:
+ return "number in XML data '%s' out of range";
+ case ERR_XML_DECL:
+ return "error making new XML declaration";
+ case ERR_XML_ELEMENT:
+ return "error making new XML element '%s'";
+ case ERR_XML_ERD:
+ return "unexpected ERD typeCode %" PRId64 " while reading XML data";
+ case ERR_XML_GONE:
+ return "ran out of XML data";
+ case ERR_XML_INPUT:
+ return "unable to read XML data from input file";
+ case ERR_XML_LEFT:
+ return "did not consume all of the XML data, '%s' left";
+ case ERR_XML_MISMATCH:
+ return "found mismatch between XML data and infoset '%s'";
+ case ERR_XML_WRITE:
+ return "error writing XML document";
+ default:
+ assert("invalid code" && 0);
+ return "unrecognized error code, shouldn't happen";
+ }
+}
+
+// print_maybe_stop - print a message and maybe stop the program
+
+static void
+print_maybe_stop(const Error *err, int status)
+{
+ const int errnum = 0;
+ const char *format = "%s";
+ const char *msg = error_message(err->code);
+
+ switch (err->code)
+ {
+ case ERR_FILE_OPEN:
+ case ERR_FIXED_VALUE:
+ case ERR_INFOSET_READ:
+ case ERR_INFOSET_WRITE:
+ case ERR_STRTOBOOL:
+ case ERR_STRTOD_ERRNO:
+ case ERR_STRTOI_ERRNO:
+ case ERR_STRTONUM_EMPTY:
+ case ERR_STRTONUM_NOT:
+ case ERR_STRTONUM_RANGE:
+ case ERR_XML_ELEMENT:
+ case ERR_XML_LEFT:
+ case ERR_XML_MISMATCH:
+ error(status, errnum, msg, err->s);
+ break;
+ case ERR_CHOICE_KEY:
+ case ERR_PARSE_BOOL:
+ case ERR_XML_ERD:
+ error(status, errnum, msg, err->d64);
+ break;
+ default:
+ error(status, errnum, format, msg);
+ break;
+ }
+}
+
+// get_diagnostics - get pointer to validation diagnostics
+
+Diagnostics *
+get_diagnostics(void)
+{
+ static Diagnostics diagnostics;
+ return &diagnostics;
+}
+
+// add_diagnostic - add a new error to validation diagnostics
+
+bool
+add_diagnostic(Diagnostics *diagnostics, const Error *error)
+{
+ if (diagnostics && error)
+ {
+ if (diagnostics->length < LIMIT_DIAGNOSTICS)
+ {
+ Error *err = &diagnostics->array[diagnostics->length++];
+ err->code = error->code;
+ err->s = error->s;
+ return true;
+ }
+ }
+ return false;
+}
+
+// print_diagnostics - print any validation diagnostics
+
+void
+print_diagnostics(const Diagnostics *diagnostics)
+{
+ if (diagnostics)
+ {
+ for (size_t i = 0; i < diagnostics->length; i++)
+ {
+ const Error *error = &diagnostics->array[i];
+ print_maybe_stop(error, 0);
+ }
+ }
+}
+
+// continue_or_exit - print and exit if an error occurred or continue otherwise
+
+void
+continue_or_exit(const Error *error)
+{
+ if (error)
Review comment:
This if statement is a stylistic practice for easier reading and
maintainability (opinion may vary), not a FPGA thing.
##########
File path:
daffodil-tdml-lib/src/main/scala/org/apache/daffodil/tdml/processor/TDMLDFDLProcessor.scala
##########
@@ -124,7 +124,10 @@ trait TDMLResult {
def isValidationError: Boolean
def isProcessingError: Boolean
def getDiagnostics: Seq[Diagnostic]
-
+ /**
+ * Deletes any temporary files that have been generated
+ */
+ def cleanUp(): Unit
Review comment:
I agree, added a no-op implementation.
##########
File path: daffodil-cli/src/main/scala/org/apache/daffodil/Main.scala
##########
@@ -502,11 +502,47 @@ class CLIConf(arguments: Array[String]) extends
scallop.ScallopConf(arguments)
val info = tally(descr = "increment test result information output level,
one level for each -i")
}
+ // Generate Subcommand Options
+ object generate extends scallop.Subcommand("generate") {
+ descr("generate <language> code from a DFDL schema")
+
+ banner("""|Usage: daffodil [GLOBAL_OPTS] generate <language>
[SUBCOMMAND_OPTS]
+ |""".stripMargin)
+ shortSubcommandsHelp()
+ footer("""|
+ |Run 'daffodil generate <language> --help' for subcommand
specific options""".stripMargin)
+
+ object c extends scallop.Subcommand("c") {
Review comment:
Well, the C code generator creates C code that any standard C99, C11, or
C17 compiler can compile without any warning messages. It's just that we try
to make as few OS calls in the C code as possible, including avoiding any calls
to malloc and free with the exception of the Mini-XML library's own malloc/free
calls. If someone really needs to generate C code that is thread-safe but
requires calls to malloc/free, then let's make another code generator and add a
new name for it like "c+" (that is, thread-safe C that falls in between plain
embeddable-safe C and pure C++).
##########
File path:
daffodil-runtime2/src/main/scala/org/apache/daffodil/runtime2/CodeGenerator.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import java.io.File
+import java.nio.file.FileSystems
+import java.nio.file.Files
+import java.nio.file.Paths
+import java.util.Collections
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.api.Diagnostic
+import org.apache.daffodil.dsom.Root
+import org.apache.daffodil.dsom.SchemaDefinitionError
+import org.apache.daffodil.runtime2.generators.CodeGeneratorState
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.xml.RefQName
+
+/**
+ * We need a mutux object for exclusive access to a code block
+ */
+private object mutex {}
+
+/**
+ * Generates and compiles C source files from a DFDL schema encapsulated in
the parameter.
+ * Implements the DFDL.CodeGenerator trait to allow it to be called by
Daffodil code.
+ * @param root Provides the DFDL schema for code generation
+ */
+class CodeGenerator(root: Root) extends DFDL.CodeGenerator {
+ // Used by compileCode and pickCompiler methods
+ private lazy val isWindows =
System.getProperty("os.name").toLowerCase().startsWith("windows")
+ // Used by WithDiagnostics methods
+ private var diagnostics: Seq[Diagnostic] = Nil
+ private var errorStatus: Boolean = false
+
+ /**
+ * Writes C source files into a "c" subdirectory of the given output
directory.
+ * Removes the "c" subdirectory if it existed before. Returns the "c"
subdirectory.
+ */
+ override def generateCode(rootNS: Option[RefQName], outputDirArg: String):
os.Path = {
+ // Get the paths of the output directory and its code subdirectory
+ val outputDir = os.Path(Paths.get(outputDirArg).toAbsolutePath)
+ val codeDir = outputDir/"c"
+
+ // Ensure our output directory exists while our code subdirectory does not
+ os.makeDir.all(outputDir)
+ os.remove.all(codeDir)
+
+ // Copy our resource directory and all its C source files to our code
subdirectory
+ // (using synchronized to avoid calling FileSystems.newFileSystem
concurrently)
+ val resourceUri = Misc.getRequiredResource("/c")
Review comment:
Good point. I've made the necessary file moves and changes.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/dsom/ElementBase.scala
##########
@@ -354,6 +354,38 @@ trait ElementBase
} else DataValue.NoValue
}
+ /**
+ * Is either DataValue[AnyRef] or DataValue.NoValue.
+ *
+ * The value will always be of the matching primitive types for the element,
and
+ * directly usable as the value of a simple-type element.
+ *
+ * When a value is used, it is created from the XSD fixed attribute of the
+ * element declaration, and that string cannot contain DFDL entities of any
kind,
+ * nor any PUA-remapped characters. This insures the fixed value can still be
+ * used for ordinary XML-schema validation outside of Daffodil/DFDL.
+ */
+ final lazy val fixedValue: DataValuePrimitiveNullable = {
+ if (hasFixedValue && (isScalar ||
isArrayWithAtLeastOneRequiredArrayElement)) {
Review comment:
You're right, I copied fixedValue from defaultValue and didn't know the
logic needed to be different. I've gone with `if (hasFixedValue &&
isSimpleType)` since it can't hurt to have both conditions just in case.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/compiler/Compiler.scala
##########
@@ -108,6 +109,27 @@ final class ProcessorFactory private(
override def onPath(xpath: String) = sset.onPath(xpath)
+ override def forLanguage(language: String): DFDL.CodeGenerator = {
+ Assert.usage(!isError)
+
+ // Do a poor man's pluggable code generator implementation - we can replace
+ // it after we observe how the validator SPI evolves and wait for our
+ // requirements to become clearer
+ val className = language match {
+ case "c" => "org.apache.daffodil.runtime2.CodeGenerator"
+ case _ => s"code generator; source language $language is not supported"
+ }
+ import scala.language.existentials // Needed to make next line compile
+ val clazz = Try(Class.forName(className))
+ val constructor = clazz.map { _.getDeclaredConstructor(sset.root.getClass)
}
+ val tryInstance = constructor.map {
_.newInstance(sset.root).asInstanceOf[DFDL.CodeGenerator] }
+ val codeGenerator = tryInstance.recover {
+ case ex => throw new InvalidParserException("Error creating " +
className, ex)
+ }.get
+
Review comment:
If my memory is correct, trying to instantiate runtime2.CodeGenerator
directly requires a circular dependency between daffodil-core and
daffodil-runtime2 which gives sbt a problem or something like that. So we need
reflection, but this is at least simpler than a full pluggable implementation
and we can revisit this place when it's time to add another code generator.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/compiler/Compiler.scala
##########
@@ -108,6 +109,27 @@ final class ProcessorFactory private(
override def onPath(xpath: String) = sset.onPath(xpath)
+ override def forLanguage(language: String): DFDL.CodeGenerator = {
+ Assert.usage(!isError)
+
+ // Do a poor man's pluggable code generator implementation - we can replace
+ // it after we observe how the validator SPI evolves and wait for our
+ // requirements to become clearer
+ val className = language match {
+ case "c" => "org.apache.daffodil.runtime2.CodeGenerator"
+ case _ => s"code generator; source language $language is not supported"
Review comment:
Sure, I've changed line 120 to throw an InvalidParserException like line
127 does below.
##########
File path:
daffodil-runtime1/src/main/scala/org/apache/daffodil/api/DFDLParserUnparser.scala
##########
@@ -147,11 +149,41 @@ object DFDL {
@deprecated("Use arguments to Compiler.compileSource or compileFile.",
"2.6.0")
def setDistinguishedRootNode(name: String, namespace: String = null): Unit
+ /**
+ * Returns a [[DataProcessor]] to process data matching a compiled XPath
expression
+ * @param xpath XPath expression in DFDL schema that data should match
(you can use only "/" at this time)
+ */
def onPath(xpath: String): DataProcessor
+
+ /**
+ * Returns a [[CodeGenerator]] to generate code from a DFDL schema to
parse or unparse data
+ * @param language source language for generated code (you can use only
"c" at this time)
+ */
+ def forLanguage(language: String): CodeGenerator
}
- trait DataProcessor extends WithDiagnostics {
+ /**
+ * Source code generation and compilation is performed with a
language-specific [[CodeGenerator]],
+ * which must be interrogated for diagnostics to see if each call was
successful or not.
+ */
+ trait CodeGenerator extends WithDiagnostics {
+ /**
+ * Generates language-specific code from a DFDL schema to parse or unparse
data
+ * @param rootNS one of the top-level elements of the DFDL schema (if not
supplied, uses first top-level element)
+ * @param outputDir output directory in which to generate code
+ * @return path of output directory containing generated code
+ */
+ def generateCode(rootNS: Option[RefQName], outputDir: String): os.Path
Review comment:
No, the returned value is a newly created subdirectory inside the passed
outputDir. Typically (when running "daffodil generate c") we pass in the
default value "." (for the current directory) and get back "c" (for the newly
created directory). A code generator for another language would return its
language's short name such as "vhld" instead. When running tests we pass in a
new temporary directory as outputDir and delete that temporary directory
following the tests so everything gets cleaned up.
##########
File path: daffodil-lib/src/main/resources/org/apache/daffodil/xsd/tdml.xsd
##########
@@ -101,6 +101,7 @@
<simpleType name="implementationItem">
<restriction base="xs:token">
<enumeration value="daffodil"/>
+ <enumeration value="daffodil-runtime2"/>
<enumeration value="ibm"/>
Review comment:
Since we hardcode these values in, let's table extensibility as a future
"nice to have" feature. By the way, one of my runtime2 TODOs is to simplify
how TDML tests specify which implementation they work with. I'll say more in
my response to the next comment, but my thought is that perhaps TDML tests can
just put their implementations in their "defaultImplementations" /
"implementations" attributes and the TDML runner can automatically run all the
implementations it knows how to call (e.g., implementations="daffodil
daffodil-runtime2" would result in calling both runtime1 and runtime2 on the
same test case without having to set a tunable
tdmlImplementation="daffodil-runtime2" as well).
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/dsom/ElementBase.scala
##########
@@ -354,6 +354,38 @@ trait ElementBase
} else DataValue.NoValue
}
+ /**
+ * Is either DataValue[AnyRef] or DataValue.NoValue.
+ *
+ * The value will always be of the matching primitive types for the element,
and
+ * directly usable as the value of a simple-type element.
+ *
+ * When a value is used, it is created from the XSD fixed attribute of the
+ * element declaration, and that string cannot contain DFDL entities of any
kind,
+ * nor any PUA-remapped characters. This insures the fixed value can still be
+ * used for ordinary XML-schema validation outside of Daffodil/DFDL.
+ */
+ final lazy val fixedValue: DataValuePrimitiveNullable = {
+ if (hasFixedValue && (isScalar ||
isArrayWithAtLeastOneRequiredArrayElement)) {
+ val dv = {
Review comment:
Yes, I changed dv to fv.
##########
File path:
daffodil-propgen/src/main/resources/org/apache/daffodil/xsd/dafext.xsd
##########
@@ -504,6 +504,14 @@
</xs:documentation>
</xs:annotation>
</xs:element>
+ <xs:element name="tdmlImplementation" type="xs:string"
default="daffodil" minOccurs="0">
+ <xs:annotation>
+ <xs:documentation>
+ TDMLDFDLProcessorFactory implementation to use when running TDML
tests.
+ Allowed values are "daffodil" (default), "daffodil-runtime2",
and "ibm".
Review comment:
In fact, one of my runtime2 TODOs is to look for a way to run at least
some of Daffodil's existing tests with both runtime1 and runtime2. The TODO is
to figure out the smallest change we possibly could make to TDML Runner, Scala
test classes, or TDML files in order to verify that some tests work on both
runtime1 and runtime2. Currently Daffodil never runs multiple TDML
implementations in parallel or serially on the same tests; perhaps we could
make Daffodil do so if the TDML file has a defaultImplementations or
implementations attribute listing multiple implementations.
##########
File path: daffodil-runtime2/src/main/resources/c/Makefile
##########
@@ -0,0 +1,55 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Here's how to compile the C sources into a program for running
+# parse/unparse tests (.dat <-> .xml)
+
+PROGRAM = ./daffodil
+HEADERS = libcli/*.h libruntime/*.h
+SOURCES = libcli/*.c libruntime/*.c
+INCLUDES = -I libcli -I libruntime
+CFLAGS = -g -Wall -Wextra
+LIBS = -lmxml
+
+$(PROGRAM): $(HEADERS) $(SOURCES)
+ $(CC) $(CFLAGS) $(INCLUDES) $(SOURCES) $(LIBS) -o $(PROGRAM)
+
+# Here's how to run parse/unparse tests (.dat <-> .xml, although you
+# will need to create the .dat and .xml files first)
+
+PARSE_DAT = parse.dat
+UNPARSE_XML = unparse.xml
+
+clean:
+ rm -f $(PROGRAM) test_$(PARSE_DAT) test_$(UNPARSE_XML)
+
+tests: parse-test unparse-test
+
+parse-test: $(PROGRAM)
+ $(PROGRAM) parse $(PARSE_DAT) -o test_$(UNPARSE_XML)
+ xmldiff $(UNPARSE_XML) test_$(UNPARSE_XML)
+
+unparse-test: $(PROGRAM)
+ $(PROGRAM) unparse $(UNPARSE_XML) -o test_$(PARSE_DAT)
+ diff $(PARSE_DAT) test_$(PARSE_DAT)
Review comment:
Yes, the code generator puts this Makefile in the newly created "c"
subdirectory for easier manual testing by a developer. Typically you'd be in a
test/resources/... directory containing TDML & data files and the next thing
you type after generating code would be to copy a pair of .dat and .xml from
the current directory into the "c" subdirectory as well:
```shell
$ cd daffodil-runtime2/src/test/resources/org/apache/daffodil/runtime2
$ daffodil generate c -s nested.dfdl.xsd -r NestedUnion
$ cp nested_union_parse_2.dat c/parse.dat
$ cp nested_union_unparse_2.xml c/unparse.xml
$ cd c
$ make
cc -g -Wall -Wextra -I libcli -I libruntime libcli/*.c libruntime/*.c -lmxml
-o ./daffodil
$ make tests
./daffodil parse parse.dat -o test_unparse.xml
xmldiff unparse.xml test_unparse.xml
./daffodil unparse unparse.xml -o test_parse.dat
diff parse.dat test_parse.dat
```
Running the TDML tests with Daffodil performs the same actions a lot faster,
so you'd use the Makefile to develop, test, or debug the C files only if needed.
##########
File path: daffodil-runtime2/src/main/resources/c/libcli/daffodil_argp.c
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "daffodil_argp.h"
+#include <argp.h> // for argp_state, argp_error, error_t, argp_parse,
ARGP_ERR_UNKNOWN, ARGP_IN_ORDER, ARGP_KEY_ARG, argp, argp_option, ARGP_KEY_END
+#include <stdio.h> // for sprintf, NULL
+#include <stdlib.h> // for putenv
+#include <string.h> // for strlen, strcmp
+
+// Initialize our "daffodil parse" CLI options
+
+struct daffodil_parse_cli daffodil_parse = {
+ "xml", // default infoset type
+ "-", // default infile
+ "-", // default outfile
+};
+
+static const struct argp_option parse_options[] = {
+ {"infoset-type", 'I', "<infoset_type>", 0, "Infoset type to output. Must
be one of 'xml' or 'null'", 0},
+
+ {"output", 'o', "<file>", 0,
+ "Write output to a given file. If not given or is -, output is written to
"
+ "stdout",
+ 0},
+
+ {0}};
+
+static error_t parse_handler(int key, char *arg, struct argp_state *state);
+
+static const char parse_args_doc[] = "[infile]";
+
+static const char parse_doc[] = "\n"
+ "Parse a file using a DFDL schema\n"
+ "\n"
+ "Parse Options:"
+ "\v"
+ " Trailing arguments:\n"
+ " infile (not required) input file to
parse. "
+ "If not specified, or a value of -, reads from
stdin";
+
+static const struct argp parse_argp = {
Review comment:
We use only libruntime in our FPGA, not this CLI. Daffodil uses this
CLI to run TDML tests with runtime2. I made the CLI user-friendly for users
just in case but I'm expecting the only users to be developers
editing/debugging the C code and Daffodil's runtime2 TDML implementation.
##########
File path: .github/workflows/main.yml
##########
@@ -25,30 +25,64 @@ jobs:
matrix:
java_version: [ 8, 11, 16 ]
scala_version: [ 2.12.11 ]
- os: [ 'ubuntu-latest', 'windows-latest' ]
- env:
- SBT: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }} coverage
- SBTNOCOV: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }}
+ os: [ ubuntu-20.04, windows-2019 ]
+ include:
+ - os: ubuntu-20.04
+ shell: bash
+ - os: windows-2019
+ shell: msys2 {0}
runs-on: ${{ matrix.os }}
defaults:
run:
- shell: bash
-
+ shell: ${{ matrix.shell }}
+ env:
+ SBT: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }} coverage
+ SBTNOCOV: sbt -J-Xms1024m -J-Xmx5120m -J-XX:ReservedCodeCacheSize=512m
-J-XX:MaxMetaspaceSize=1024m ++${{ matrix.scala_version }}
steps:
############################################################
# Setup
############################################################
- - name: Checkout Repository
- uses: actions/[email protected]
+ - name: Install Dependencies (Linux)
+ if: runner.os == 'Linux'
+ run: sudo apt-get install -y libmxml-dev
+
+ - name: Install Dependencies (Windows)
+ if: runner.os == 'Windows'
+ uses: msys2/setup-msys2@v2
+ with:
+ install: gcc libargp-devel make pkgconf
Review comment:
I'm not certain what the best answer to your question is. The argp
library is widely available, so we don't distribute it with Daffodil in any way
or form (no sources, no lib.so or lib.a archive, no executables linked with
it). When we build a Daffodil release or run Daffodil on a TDML file, both
Daffodil's tests ("sbt test") and Daffodil's CLI ("daffodil test \<runtime2
TDML file\>") need to call daffodil_argp.c's CLI (Daffodil generates the C
files, compiles and links the C files to build a temporary C executable, calls
the C executable as a subprocess, waits for it to exit, and examines whether
the C executable parsed or unparsed the test data successfully). If the test
passes, Daffodil immediately deletes all the files it had generated, compiled,
written, executed, and read, leaving nothing behind. If you are willing to
build a Daffodil release on Windows without test-building the C files and
running the runtime2 tests, then you wouldn't need to install the libargp lib
rary on Windows. On Fedora and Ubuntu, libargp is included in glibc so it's
always available.
When the user calls Daffodil's CLI generate subcommand ("daffodil generate c
-s \<DFDL schema\>"), Daffodil also writes out all the C files to an output
directory along with a Makefile to help the user compile the C files. In this
case, the libargp dependency still is optional because the user can use only
the libruntime2 files in their own application with their own main function.
That is, the user is free (in fact, expected to) to ignore the libcli files and
use only the libruntime2 files to parse a stream to in-memory data or unparse
in-memory data to a stream. Our FPGA softcore processor uses only the
libruntime2 files.
##########
File path: daffodil-runtime2/src/main/resources/c/Makefile
##########
@@ -0,0 +1,55 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Here's how to compile the C sources into a program for running
+# parse/unparse tests (.dat <-> .xml)
+
+PROGRAM = ./daffodil
+HEADERS = libcli/*.h libruntime/*.h
+SOURCES = libcli/*.c libruntime/*.c
+INCLUDES = -I libcli -I libruntime
+CFLAGS = -g -Wall -Wextra
+LIBS = -lmxml
+
+$(PROGRAM): $(HEADERS) $(SOURCES)
+ $(CC) $(CFLAGS) $(INCLUDES) $(SOURCES) $(LIBS) -o $(PROGRAM)
+
+# Here's how to run parse/unparse tests (.dat <-> .xml, although you
+# will need to create the .dat and .xml files first)
+
+PARSE_DAT = parse.dat
+UNPARSE_XML = unparse.xml
+
+clean:
+ rm -f $(PROGRAM) test_$(PARSE_DAT) test_$(UNPARSE_XML)
+
+tests: parse-test unparse-test
+
+parse-test: $(PROGRAM)
+ $(PROGRAM) parse $(PARSE_DAT) -o test_$(UNPARSE_XML)
+ xmldiff $(UNPARSE_XML) test_$(UNPARSE_XML)
+
+unparse-test: $(PROGRAM)
+ $(PROGRAM) unparse $(UNPARSE_XML) -o test_$(PARSE_DAT)
+ diff $(PARSE_DAT) test_$(PARSE_DAT)
+
+# You will need the Mini-XML library and xmldiff - here's how to
+# install both in Ubuntu 20.04
+
+deps:
+ sudo apt install libmxml-dev xmldiff
Review comment:
I can move the sudo apt command into the comment if you want. I figured
anyone would read the target first and wouldn't call it unless they really
wanted to run that command.
##########
File path: daffodil-runtime2/src/main/resources/examples/NestedUnion.c
##########
@@ -0,0 +1,442 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
Review comment:
I want to keep the example files as close to the resource C files as
possible. They do go in the daffodil-runtime2 jar because of their location in
src/main/resources/examples, but we could exclude them if necessary. Their
chief (only) use is for comparisons (diffs) and code reviews (diffs again). I
have to manually update them if I make a change to the generator. I hadn't
considered writing tests that validate these match generated code. The
comparison would have to ignore the copyright header and the first include
which always differ right now. I doubt such tests would have much utility or
usefulness other than to ensure that the examples are up to date.
##########
File path: daffodil-runtime2/src/main/resources/c/libcli/xml_writer.c
##########
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "xml_writer.h"
+#include <assert.h> // for assert
+#include <mxml.h> // for mxmlNewOpaquef, mxml_node_t, mxmlElementSetAttr,
mxmlGetOpaque, mxmlNewElement, mxmlDelete, mxmlGetElement, mxmlNewXML,
mxmlSaveFile, MXML_NO_CALLBACK
+#include <stdbool.h> // for bool
+#include <stdint.h> // for int16_t, int32_t, int64_t, int8_t, uint16_t,
uint32_t, uint64_t, uint8_t
+#include <stdio.h> // for NULL
+#include <string.h> // for strcmp
+#include "errors.h" // for Error, ERR_XML_DECL, ERR_XML_ELEMENT,
ERR_XML_WRITE, LIMIT_XML_NESTING, Error::(anonymous)
+#include "stack.h" // for stack_is_empty, stack_pop, stack_push, stack_top,
stack_init
+
+// Push new XML document on stack (note the stack is stored in a
+// static array which could overflow and stop the program; it also
+// means none of those functions are thread-safe)
+
+static const Error *
+xmlStartDocument(XMLWriter *writer)
+{
+ static mxml_node_t *array[LIMIT_XML_NESTING];
+ stack_init(&writer->stack, array, LIMIT_XML_NESTING);
+
+ mxml_node_t *xml = mxmlNewXML("1.0");
+ if (xml)
+ {
+ stack_push(&writer->stack, xml);
+ return NULL;
+ }
+ else
+ {
+ static Error error = {ERR_XML_DECL, {NULL}};
+ return &error;
+ }
+}
+
+// Pop completed XML document off stack and write it to stream (note
+// stack underflow will stop program)
+
+static const Error *
+xmlEndDocument(XMLWriter *writer)
+{
+ mxml_node_t *xml = stack_pop(&writer->stack);
+ assert(stack_is_empty(&writer->stack));
+
+ int status = mxmlSaveFile(xml, writer->stream, MXML_NO_CALLBACK);
+ if (status < 0)
+ {
+ static Error error = {ERR_XML_WRITE, {NULL}};
+ return &error;
+ }
+ mxmlDelete(xml);
+ return NULL;
+}
+
+// Push new complex element on stack (note stack overflow will stop
+// program)
+
+static const Error *
+xmlStartComplex(XMLWriter *writer, const InfosetBase *base)
+{
+ mxml_node_t *parent = stack_top(&writer->stack);
+ const char * name = get_erd_name(base->erd);
+ const char * xmlns = get_erd_xmlns(base->erd);
+ mxml_node_t *complex = mxmlNewElement(parent, name);
+ if (xmlns)
+ {
+ const char *ns = get_erd_ns(base->erd);
+ mxmlElementSetAttr(complex, xmlns, ns);
+ }
+ stack_push(&writer->stack, complex);
+ return NULL;
+}
+
+// Pop completed complex element off stack (note stack underflow will
+// stop program)
+
+static const Error *
+xmlEndComplex(XMLWriter *writer, const InfosetBase *base)
+{
+ mxml_node_t *complex = stack_pop(&writer->stack);
+
+ const char *name_from_xml = mxmlGetElement(complex);
+ const char *name_from_erd = get_erd_name(base->erd);
+ assert(strcmp(name_from_xml, name_from_erd) == 0);
+
+ return NULL;
+}
+
+// Fix a real number to conform to xsd:float syntax if needed
+
+static void
+fixNumberIfNeeded(const char *text)
+{
+ if (text[0] == 'N' && text[1] == 'A')
Review comment:
Perhaps we should check all that too, but this is a static helper
function that is callable only inside this file and I know it's being passed a
string formatted by printf's %G specifier, so I'm only checking for "NA" before
assuming I've got a "NAN" that needs to be replaced with "NaN".
##########
File path: daffodil-runtime2/src/main/resources/c/libruntime/errors.c
##########
@@ -0,0 +1,203 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "errors.h"
+#include <assert.h> // for assert
+#include <error.h> // for error
+#include <inttypes.h> // for PRId64
+#include <stdbool.h> // for bool, false, true
+#include <stdio.h> // for NULL, feof, ferror, FILE, size_t
+#include <stdlib.h> // for EXIT_FAILURE
+
+// error_message - return an internationalized error message
+
+static const char *
+error_message(enum ErrorCode code)
+{
+ switch (code)
+ {
+ case ERR_CHOICE_KEY:
+ return "no match between choice dispatch key %" PRId64 " and any
branch key";
+ case ERR_FILE_CLOSE:
+ return "error closing file";
+ case ERR_FILE_FLUSH:
+ return "error flushing stream to file";
+ case ERR_FILE_OPEN:
+ return "error opening file '%s'";
+ case ERR_FIXED_VALUE:
+ return "value of element '%s' does not match value of its "
+ "'fixed' attribute";
+ case ERR_INFOSET_READ:
+ return "cannot read infoset type '%s'";
+ case ERR_INFOSET_WRITE:
+ return "cannot write infoset type '%s'";
+ case ERR_PARSE_BOOL:
+ return "error parsing binary value %" PRId64 " as either true or
false";
+ case ERR_STACK_EMPTY:
+ return "stack empty, stopping program";
+ case ERR_STACK_OVERFLOW:
+ return "stack overflow, stopping program";
+ case ERR_STACK_UNDERFLOW:
+ return "stack underflow, stopping program";
+ case ERR_STREAM_EOF:
+ return "EOF in stream, stopping program";
+ case ERR_STREAM_ERROR:
+ return "error in stream, stopping program";
+ case ERR_STRTOBOOL:
+ return "error converting XML data '%s' to boolean";
+ case ERR_STRTOD_ERRNO:
+ return "error converting XML data '%s' to number";
+ case ERR_STRTOI_ERRNO:
+ return "error converting XML data '%s' to integer";
+ case ERR_STRTONUM_EMPTY:
+ return "found no number in XML data '%s'";
+ case ERR_STRTONUM_NOT:
+ return "found non-number characters in XML data '%s'";
+ case ERR_STRTONUM_RANGE:
+ return "number in XML data '%s' out of range";
+ case ERR_XML_DECL:
+ return "error making new XML declaration";
+ case ERR_XML_ELEMENT:
+ return "error making new XML element '%s'";
+ case ERR_XML_ERD:
+ return "unexpected ERD typeCode %" PRId64 " while reading XML data";
+ case ERR_XML_GONE:
+ return "ran out of XML data";
+ case ERR_XML_INPUT:
+ return "unable to read XML data from input file";
+ case ERR_XML_LEFT:
+ return "did not consume all of the XML data, '%s' left";
+ case ERR_XML_MISMATCH:
+ return "found mismatch between XML data and infoset '%s'";
+ case ERR_XML_WRITE:
+ return "error writing XML document";
+ default:
+ assert("invalid code" && 0);
+ return "unrecognized error code, shouldn't happen";
+ }
+}
+
+// print_maybe_stop - print a message and maybe stop the program
+
+static void
+print_maybe_stop(const Error *err, int status)
+{
+ const int errnum = 0;
+ const char *format = "%s";
+ const char *msg = error_message(err->code);
+
+ switch (err->code)
+ {
+ case ERR_FILE_OPEN:
+ case ERR_FIXED_VALUE:
+ case ERR_INFOSET_READ:
+ case ERR_INFOSET_WRITE:
+ case ERR_STRTOBOOL:
+ case ERR_STRTOD_ERRNO:
+ case ERR_STRTOI_ERRNO:
+ case ERR_STRTONUM_EMPTY:
+ case ERR_STRTONUM_NOT:
+ case ERR_STRTONUM_RANGE:
+ case ERR_XML_ELEMENT:
+ case ERR_XML_LEFT:
+ case ERR_XML_MISMATCH:
+ error(status, errnum, msg, err->s);
+ break;
+ case ERR_CHOICE_KEY:
+ case ERR_PARSE_BOOL:
+ case ERR_XML_ERD:
+ error(status, errnum, msg, err->d64);
+ break;
+ default:
+ error(status, errnum, format, msg);
+ break;
+ }
+}
+
+// get_diagnostics - get pointer to validation diagnostics
+
+Diagnostics *
+get_diagnostics(void)
+{
+ static Diagnostics diagnostics;
+ return &diagnostics;
+}
+
+// add_diagnostic - add a new error to validation diagnostics
+
+bool
+add_diagnostic(Diagnostics *diagnostics, const Error *error)
+{
+ if (diagnostics && error)
+ {
+ if (diagnostics->length < LIMIT_DIAGNOSTICS)
Review comment:
Mike initially suggested stopping the program but then agreed to let the
caller decide. Now add_diagnostic returns false to warn the caller if a
diagnostic couldn't be added successfully. The caller can then stop the
program or continue if it wants. In practice, our code will accumulate a
maximum of 100 validation errors (element x doesn't match fixed value), drop
the rest, and print the first 100 validation errors at the end of the parse so
the user will know whether the data had any parser error (more important to
show to user) as well as whether the data had lots of validation errors (less
important to show to user).
##########
File path:
daffodil-runtime2/src/main/scala/org/apache/daffodil/runtime2/CodeGenerator.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import java.io.File
+import java.nio.file.FileSystems
+import java.nio.file.Files
+import java.nio.file.Paths
+import java.util.Collections
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.api.Diagnostic
+import org.apache.daffodil.dsom.Root
+import org.apache.daffodil.dsom.SchemaDefinitionError
+import org.apache.daffodil.runtime2.generators.CodeGeneratorState
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.xml.RefQName
+
+/**
+ * We need a mutux object for exclusive access to a code block
+ */
+private object mutex {}
+
+/**
+ * Generates and compiles C source files from a DFDL schema encapsulated in
the parameter.
+ * Implements the DFDL.CodeGenerator trait to allow it to be called by
Daffodil code.
+ * @param root Provides the DFDL schema for code generation
+ */
+class CodeGenerator(root: Root) extends DFDL.CodeGenerator {
+ // Used by compileCode and pickCompiler methods
+ private lazy val isWindows =
System.getProperty("os.name").toLowerCase().startsWith("windows")
+ // Used by WithDiagnostics methods
+ private var diagnostics: Seq[Diagnostic] = Nil
+ private var errorStatus: Boolean = false
Review comment:
Good point. I've updated the scaladoc to say:
```
* Note: Also implements WithDiagnostics trait with mutable state which
means you need
* to create a mew CodeGenerator each time you generate code.
```
##########
File path:
daffodil-runtime2/src/main/scala/org/apache/daffodil/runtime2/CodeGenerator.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import java.io.File
+import java.nio.file.FileSystems
+import java.nio.file.Files
+import java.nio.file.Paths
+import java.util.Collections
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.api.Diagnostic
+import org.apache.daffodil.dsom.Root
+import org.apache.daffodil.dsom.SchemaDefinitionError
+import org.apache.daffodil.runtime2.generators.CodeGeneratorState
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.xml.RefQName
+
+/**
+ * We need a mutux object for exclusive access to a code block
+ */
+private object mutex {}
+
+/**
+ * Generates and compiles C source files from a DFDL schema encapsulated in
the parameter.
+ * Implements the DFDL.CodeGenerator trait to allow it to be called by
Daffodil code.
+ * @param root Provides the DFDL schema for code generation
+ */
+class CodeGenerator(root: Root) extends DFDL.CodeGenerator {
+ // Used by compileCode and pickCompiler methods
+ private lazy val isWindows =
System.getProperty("os.name").toLowerCase().startsWith("windows")
Review comment:
Changed to call isWin rather than isWindows. Note that daffodil-cli
uses Util.isWindows (with same definition as above) in many places; might want
to make daffodil-cli use isWin for consistency too.
##########
File path:
daffodil-runtime2/src/main/scala/org/apache/daffodil/runtime2/Runtime2DataProcessor.scala
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import java.io.File
+import java.io.InputStream
+import java.io.OutputStream
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.api.DaffodilTunables
+import org.apache.daffodil.api.DataLocation
+import org.apache.daffodil.api.ValidationMode
+import org.apache.daffodil.externalvars.Binding
+import org.apache.daffodil.processors.Failure
+import org.apache.daffodil.processors.ProcessorResult
+import org.apache.daffodil.processors.Success
+import org.apache.daffodil.processors.VariableMap
+import org.apache.daffodil.processors.WithDiagnosticsImpl
+import org.apache.daffodil.processors.parsers.ParseError
+import org.apache.daffodil.processors.unparsers.UnparseError
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.util.Maybe.Nope
+
+/**
+ * Effectively a scala proxy object that does its work via the underlying
C-code.
+ * Will need to consider how to use features of underlying C-code to get
infoset,
+ * walk infoset, generate XML for use by TDML tests.
+ */
+class Runtime2DataProcessor(executableFile: os.Path) extends
DFDL.DataProcessorBase {
Review comment:
Yes, we call this file only when the TDML runner is using the
"daffodil-runtime2" implementation. It shouldn't be necessary to change the
???'s since Scala already throws an "unsupported" exception if anything calls
???. I checked the IBM runner and it does have a bunch of ???'s there as well:
```rg
src/main/scala/org/apache/daffodil/tdml/processor/IBM_DFDL.scala
74: override def getSchemaLocation: URI = ???
143: @Deprecated def setTunables(tunables: Map[String, String]): Unit = ???
148: @Deprecated def setDistinguishedRootNode(name: String, namespace:
String): Unit = ???
345: override def withDebugger(db: Object): IBMTDMLDFDLProcessor = ???
347: if (onOff) ???
368: @Deprecated override def setExternalDFDLVariables(externalVarBindings:
Seq[Binding]): Unit = ???
369: @Deprecated override def setDebugging(onOff: Boolean): Unit = ???
370: @Deprecated override def setTracing(onOff: Boolean): Unit = ???
371: @Deprecated override def setValidationMode(validationMode:
ValidationMode.Type): Unit = ???
372: @Deprecated override def setDebugger(db: AnyRef): Unit = ???
```
##########
File path:
daffodil-runtime2/src/main/scala/org/apache/daffodil/runtime2/Runtime2CodeGenerator.scala
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import org.apache.daffodil.grammar.Gram
+import org.apache.daffodil.grammar.Prod
+import org.apache.daffodil.grammar.RootGrammarMixin
+import org.apache.daffodil.grammar.SeqComp
+import org.apache.daffodil.grammar.primitives.BinaryBoolean
+import org.apache.daffodil.grammar.primitives.BinaryDouble
+import org.apache.daffodil.grammar.primitives.BinaryFloat
+import org.apache.daffodil.grammar.primitives.BinaryIntegerKnownLength
+import org.apache.daffodil.grammar.primitives.CaptureContentLengthEnd
+import org.apache.daffodil.grammar.primitives.CaptureContentLengthStart
+import org.apache.daffodil.grammar.primitives.CaptureValueLengthEnd
+import org.apache.daffodil.grammar.primitives.CaptureValueLengthStart
+import org.apache.daffodil.grammar.primitives.ElementCombinator
+import org.apache.daffodil.grammar.primitives.ElementParseAndUnspecifiedLength
+import org.apache.daffodil.grammar.primitives.OrderedSequence
+import org.apache.daffodil.grammar.primitives.ScalarOrderedSequenceChild
+import org.apache.daffodil.grammar.primitives.SpecifiedLengthImplicit
+import org.apache.daffodil.runtime2.generators.BinaryBooleanCodeGenerator
+import org.apache.daffodil.runtime2.generators.BinaryFloatCodeGenerator
+import
org.apache.daffodil.runtime2.generators.BinaryIntegerKnownLengthCodeGenerator
+import org.apache.daffodil.runtime2.generators.CodeGeneratorState
+import
org.apache.daffodil.runtime2.generators.ElementParseAndUnspecifiedLengthCodeGenerator
+import org.apache.daffodil.runtime2.generators.OrderedSequenceCodeGenerator
+import org.apache.daffodil.runtime2.generators.SeqCompCodeGenerator
+import org.apache.daffodil.util.Misc
+
+import scala.annotation.tailrec
+
+object Runtime2CodeGenerator
+ extends BinaryBooleanCodeGenerator
+ with BinaryIntegerKnownLengthCodeGenerator
+ with BinaryFloatCodeGenerator
+ with ElementParseAndUnspecifiedLengthCodeGenerator
+ with OrderedSequenceCodeGenerator
+ with SeqCompCodeGenerator {
+
+ @tailrec
+ def generateCode(gram: Gram, state: CodeGeneratorState): Unit = {
+ gram match {
+ case g: RootGrammarMixin =>
Runtime2CodeGenerator.generateCode(g.documentElement, state)
+ case g: Prod if (g.guard) => Runtime2CodeGenerator.generateCode(g.gram,
state)
+ case g: ElementCombinator =>
Runtime2CodeGenerator.generateCode(g.subComb, state)
+ case g: SpecifiedLengthImplicit =>
Runtime2CodeGenerator.generateCode(g.eGram, state)
+ case g: ScalarOrderedSequenceChild =>
Runtime2CodeGenerator.generateCode(g.term.termContentBody, state)
+ case g: BinaryBoolean => binaryBooleanGenerateCode(g.e, state)
+ case g: BinaryDouble => binaryFloatGenerateCode(g.e, 64, state)
+ case g: BinaryFloat => binaryFloatGenerateCode(g.e, 32, state)
+ case g: BinaryIntegerKnownLength =>
binaryIntegerKnownLengthGenerateCode(g, state)
+ case g: ElementParseAndUnspecifiedLength =>
elementParseAndUnspecifiedLengthGenerateCode(g, state)
+ case g: OrderedSequence => orderedSequenceGenerateCode(g, state)
+ case g: SeqComp => seqCompGenerateCode(g, state)
+ case _: CaptureContentLengthStart => noop
+ case _: CaptureContentLengthEnd => noop
Review comment:
These lines are holdovers from when Mike and I originally implemented
the generateCode method as an abstract method of Element or Gram or whatever
the root trait was. We had to implement the generateCode methods in these
classes (RootGrammarMixin, ScalarOrderedSequenceChild, OrderedSequence, or
similar classes). Then we realized we would touch fewer files if we
implemented generateCode in its own object which knows how to walk the
document, so we changed these methods to cases in this function. However, it
looks like walking the document with cases in this function skips some places
that the abstract method forced us to implement. I would like to run some
existing TDML tests with runtime2 to make sure we don't overlook anything, but
we may not need those cases.
##########
File path: project/Dependencies.scala
##########
@@ -22,6 +22,7 @@ object Dependencies {
lazy val common = core ++ infoset ++ test
lazy val core = Seq(
+ "com.lihaoyi" %% "os-lib" % "0.7.1", // for generating C source files
Review comment:
Thanks for noticing, updated to even more recent 0.7.4 version.
##########
File path:
daffodil-tdml-lib/src/main/scala/org/apache/daffodil/tdml/TDMLRunner.scala
##########
@@ -477,7 +476,13 @@ abstract class TestCase(testCaseXML: NodeSeq, val parent:
DFDLTestSuite)
lazy val tdmlDFDLProcessorFactory: AbstractTDMLDFDLProcessorFactory = {
import scala.language.existentials
- val className =
"org.apache.daffodil.tdml.processor.TDMLDFDLProcessorFactory"
+ // tdmlImplementation is a tunable choice with three values.
+ val className = tunableObj.tdmlImplementation match {
+ // Right now daffodil and ibm use the same ProcessFactory name
+ case "daffodil" | "ibm" =>
"org.apache.daffodil.tdml.processor.TDMLDFDLProcessorFactory"
+ case "daffodil-runtime2" =>
"org.apache.daffodil.tdml.processor.Runtime2TDMLDFDLProcessorFactory"
+ case other => Assert.invariantFailed("'%s' not valid for
tdmlImplementation".format(other))
Review comment:
Also note that once this goes in, we could change ibm's ProcessFactory
to have a different name than daffodil's ProcessFactory and update this match
expression to use the new name. This would make it possible for some
developers to add ibmDFDLCrossTester's jar to their Daffodil classpath
permanently instead of having to do/undo that change each time they want to run
cross tests. I'm not sure if there is a way to set a Daffodil tunable while
running "sbt test", but if so, you could set tdmlImplementation="ibm" and then
"sbt test" would run all the cross tests. If this way works as I hope it does,
then we can start the process of adding "daffodil-runtime2" to some of the
cross tests to cover runtime2 too. I think continuing to use the
tdmlImplementation tunable would be simpler than making TDML Runner
automatically run all the TDML implementations it can find in parallel or
serially on all cross tests, although that would be the ultimate way to verify
all cross tests pass on
all platforms all the time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
