tuxji commented on a change in pull request #422:
URL: https://github.com/apache/incubator-daffodil/pull/422#discussion_r499035314
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/ElementCombinator.scala
##########
@@ -328,9 +351,21 @@ class ElementParseAndUnspecifiedLength(context:
ElementBase, eBeforeGram: Gram,
new ElementUnparserNoRep(context.erd, uSetVar)
}
}
+
+ override def generateCode(cgState: CodeGeneratorState): Unit = {
+ context.schemaDefinitionWhen(context.inputValueCalcOption.isDefined,
"Elements with inputValueCalc are not supported.")
+ context.schemaDefinitionWhen(context.outputValueCalcOption.isDefined,
"Elements with outputValueCalc are not supported.")
+ context.schemaDefinitionUnless(eBeforeGram.isEmpty, "Statements associated
with elements are not supported.")
+ context.schemaDefinitionUnless(eAfterGram.isEmpty, "Statements associated
with elements are not supported.")
+ context.schemaDefinitionUnless(repTypeElementGram.isEmpty, "dfdlx:repType
is not supported.")
+
+ val elementContentGenerator = eGram // a Gram isA ParserGenerator
+ val e = new ElementParserGenerator(context, elementContentGenerator)
+ e.generateCode(cgState)
+ }
Review comment:
Mike, please let me know if you think it would be easier to keep these
methods and remove GramRuntime2Mixin at a later time, or if you think you will
have time to pair with me to help me refactor these methods to a function in
daffodil-backend-generator-c. Unless you can point me to some example code in
Daffodil, I doubt it will be obvious to me how to walk the gram objects and
come across the corresponding ElementParseAndUnspecifiedLength gram object so I
can move the code above to a match expression inside the new function.
##########
File path: build.sbt
##########
@@ -43,6 +46,32 @@ lazy val runtime1 = Project("daffodil-runtime1",
file("daffodil-runtime1
.dependsOn(io, lib % "test->test", udf, macroLib
% "compile-internal, test-internal")
.settings(commonSettings, usesMacros)
+val runtime2StaticLib = Library("libruntime2.a")
+lazy val runtime2 = Project("daffodil-runtime2",
file("daffodil-runtime2")).configs(IntegrationTest)
+ .enablePlugins(CcPlugin)
+ .dependsOn(tdmlProc)
+ .settings(commonSettings)
+ .settings(publishArtifact in (Compile,
packageDoc) := false)
+ .settings(
+ Compile / ccTargets :=
ListSet(runtime2StaticLib),
+ Compile / cSources := Map(
+ runtime2StaticLib -> Seq(
+ baseDirectory.value / "src" / "main" / "c"
/ "common_runtime.c",
Review comment:
OK, let's choose a time when to move every file that goes into the
static lib into their own subdirectory and then perform a glob listing of that
directory in the sbt build.sbt file. I'd appreciate help from someone more
familiar with sbt anyway.
##########
File path: .github/workflows/main.yml
##########
@@ -20,81 +20,98 @@ on: [push, pull_request]
jobs:
test:
name: Java ${{ matrix.java_version }}, Scala ${{ matrix.scala_version }},
${{ matrix.os }}
- runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
- java_version: [ '8.x', '9.x', '11.x' ]
- scala_version: [ '2.12.11', '2.11.12' ]
- os: [ 'ubuntu-latest', 'windows-latest' ]
+ java_version: [ 8, 9, 11 ]
+ scala_version: [ 2.12.11, 2.11.12 ]
Review comment:
Shall I create a bug issue and remove support for Scala 2.11 in a
separate PR, then rebase my branch to catch up with asf/master again?
##########
File path: daffodil-runtime2/src/main/c/daffodil_argp.c
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "daffodil_argp.h" // for daffodil_cli, daffodil_parse_cli, ...
+#include <argp.h> // for argp_state, argp_error, error_t, argp_parse
+#include <stdio.h> // for sprintf
+#include <stdlib.h> // for putenv, NULL
+#include <string.h> // for strlen, strcmp
+
+// Initialize our "daffodil" name and version
+
+const char *argp_program_version = "Apache Daffodil (runtime2) 0.1";
+
+// Initialize our "daffodil parse" CLI options
+
+struct daffodil_parse_cli daffodil_parse = {
+ "xml", // default infoset type
+ "-", // default infile
+ "-", // default outfile
+};
+
+static const struct argp_option parse_options[] = {
+ {"infoset-type", 'I', "<infoset_type>", 0,
+ "Infoset type to output. Must be one of 'xml' or 'null'"},
+
+ {"output", 'o', "<file>", 0,
+ "Write output to a given file. If not given or is -, output is written to
"
+ "stdout"},
+
+ {0}};
+
+static error_t parse_handler(int key, char *arg, struct argp_state *state);
+
+static const char parse_args_doc[] = "[infile]";
+
+static const char parse_doc[] =
+ "\n"
+ "Parse a file using a DFDL schema\n"
+ "\n"
+ "Parse Options:"
+ "\v"
+ " Trailing arguments:\n"
+ " infile (not required) input file to parse. "
+ "If not specified, or a value of -, reads from stdin";
+
+static const struct argp parse_argp = {
+ parse_options, // array of CLI options
+ parse_handler, // function to get these CLI options
+ parse_args_doc, // short usage documentation
+ parse_doc, // long help documentation
+};
+
+// Handle callbacks to get our "daffodil parse" CLI options
+
+static error_t
+parse_handler(int key, char *arg, struct argp_state *state)
+{
+ struct daffodil_parse_cli *parse = state->input;
+
+ switch (key)
+ {
+ case 'I':
+ parse->infoset_type = arg;
+ break;
+
+ case 'o':
+ parse->outfile = arg;
+ break;
+
+ case ARGP_KEY_ARG:
+ if (state->arg_num)
+ {
+ argp_error(state, "too many arguments: %s", arg);
+ }
+ parse->infile = arg;
+ break;
+
+ default:
+ return ARGP_ERR_UNKNOWN;
+ }
+
+ return 0;
+}
+
+// Parse our "daffodil parse" command line interface
+
+static error_t
+parse_daffodil_parse_cli(struct argp_state *state)
+{
+ int argc = state->argc - state->next + 1;
+ char **argv = &state->argv[state->next - 1];
+ char * old_cmd = argv[0];
+ char new_cmd[strlen(state->name) + strlen(" parse") + 1];
+
+ sprintf(new_cmd, "%s parse", state->name);
+ argv[0] = new_cmd;
+
+ error_t status = argp_parse(&parse_argp, argc, argv, ARGP_IN_ORDER, &argc,
+ &daffodil_parse);
+
+ argv[0] = old_cmd;
+ state->next += argc - 1;
+
+ return status;
+}
+
+// Initialize our "daffodil unparse" CLI options
+
+struct daffodil_unparse_cli daffodil_unparse = {
+ "xml", // default infoset type
+ "-", // default infile
+ "-", // default outfile
+};
+
+static const struct argp_option unparse_options[] = {
+ {"infoset-type", 'I', "<infoset_type>", 0,
+ "Infoset type to unparse. Must be 'xml'"},
+
+ {"output", 'o', "<file>", 0,
+ "Write output to file. If not given or is -, output is written to "
+ "standard output"},
+
+ {0}};
+
+static error_t unparse_handler(int key, char *arg, struct argp_state *state);
+
+static const char unparse_args_doc[] = "[infile]";
+
+static const char unparse_doc[] =
+ "\n"
+ "Unparse an infoset file using a DFDL schema\n"
+ "\n"
+ "Unparse Options:"
+ "\v"
+ " Trailing arguments:\n"
+ " infile (not required) input file to unparse. If not specified, or "
+ "a value of -, reads from stdin";
+
+static const struct argp unparse_argp = {
+ unparse_options, // array of CLI options
+ unparse_handler, // function to get these CLI options
+ unparse_args_doc, // short usage documentation
+ unparse_doc, // long help documentation
+};
+
+// Handle callbacks to get our "daffodil unparse" CLI options
+
+static error_t
+unparse_handler(int key, char *arg, struct argp_state *state)
+{
+ struct daffodil_unparse_cli *unparse = state->input;
+
+ switch (key)
+ {
+ case 'I':
+ unparse->infoset_type = arg;
+ break;
+
+ case 'o':
+ unparse->outfile = arg;
+ break;
+
+ case ARGP_KEY_ARG:
+ if (state->arg_num)
+ {
+ argp_error(state, "too many arguments: %s", arg);
+ }
+ unparse->infile = arg;
+ break;
+
+ default:
+ return ARGP_ERR_UNKNOWN;
+ }
+
+ return 0;
+}
+
+// Parse our "daffodil unparse" command line interface
+
+static error_t
+parse_daffodil_unparse_cli(struct argp_state *state)
+{
+ int argc = state->argc - state->next + 1;
+ char **argv = &state->argv[state->next - 1];
+ char * old_cmd = argv[0];
+ char new_cmd[strlen(state->name) + strlen(" unparse") + 1];
+
+ sprintf(new_cmd, "%s unparse", state->name);
+ argv[0] = new_cmd;
+
+ error_t status = argp_parse(&unparse_argp, argc, argv, ARGP_IN_ORDER,
&argc,
+ &daffodil_unparse);
+
+ argv[0] = old_cmd;
+ state->next += argc - 1;
+
+ return status;
+}
+
+// Initialize our "daffodil" CLI options
+
+struct daffodil_cli daffodil_cli = {
+ DAFFODIL_NONE, // default subcommand
+ 0, // default verbosity
+};
+
+static const struct argp_option daffodil_options[] = {
+ {"verbose", 'v', 0, 0, "Increment verbosity level, one level for each -v",
+ -1},
+
+ {0}};
+
+static error_t daffodil_handler(int key, char *arg, struct argp_state *state);
+
+static const char daffodil_args_doc[] = "<subcommand> [SUBCOMMAND_OPTION...]";
+
+static const char daffodil_doc[] =
+ "\n"
+ "Global Options:"
+ "\v"
+ "Subcommands:\n"
+ " parse Parse data to a DFDL infoset\n"
+ " unparse Unparse a DFDL infoset\n"
+ "\n"
+ "Run 'daffodil <subcommand> --help' for subcommand specific options";
+
+static const struct argp daffodil_argp = {
+ daffodil_options, // array of CLI options
+ daffodil_handler, // function to get these CLI options
+ daffodil_args_doc, // short usage documentation
+ daffodil_doc, // long help documentation
+};
+
+// Handle callbacks to get our "daffodil" CLI options
+
+static error_t
+daffodil_handler(int key, char *arg, struct argp_state *state)
+{
+ struct daffodil_cli *daffodil = state->input;
Review comment:
Sure, it's a local variable so we can name it "cli" instead of
"daffodil".
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/ElementCombinator.scala
##########
@@ -280,8 +300,11 @@ case class CaptureValueLengthEnd(ctxt: ElementBase)
new CaptureEndOfValueLengthUnparser(ctxt.erd)
else
new NadaUnparser(ctxt.erd)
-}
+ override def generateCode(state: CodeGeneratorState): Unit = {
+ // Not generating code here
Review comment:
Actually, I will have to remove these generateCode() methods anyway once
I delete GramRuntime2Mixin and its generateCode() method. I'm not sure how
easy it will be to do this refactoring, but I'd like to do it sooner rather
than later.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/runtime2/GeneratedCodeCompiler.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import org.apache.commons.io.FileUtils
+import org.apache.daffodil.compiler.ProcessorFactory
+import org.apache.daffodil.dsom.SchemaDefinitionError
+import org.apache.daffodil.runtime2.generators.CodeGeneratorState
+import org.apache.daffodil.util.Misc
+import os.Pipe
+
+class GeneratedCodeCompiler(pf: ProcessorFactory) {
+ private var executableFile: os.Path = _
+ private lazy val isWindows =
System.getProperty("os.name").toLowerCase().startsWith("windows")
+
+ /**
+ * Compiles and links generated C code with runtime2 library to
+ * build an executable file.
+ */
+ def compile(rootElementName: String, codeGeneratorState:
CodeGeneratorState): Unit = {
+ val compiler = "cc"
+ val location = Option(this.getClass.getProtectionDomain.getCodeSource)
flatMap (x => Option(x.getLocation))
+ val wd = if (os.exists(os.pwd/"daffodil-runtime2"))
+ os.pwd
+ else if (os.exists(os.pwd/os.up/"daffodil-runtime2"))
+ os.pwd/os.up
+ else if (location.isDefined)
+ os.Path(FileUtils.toFile(location.get))/os.up/os.up
+ else
+ os.pwd
+ val includeDir = if (os.exists(wd/"include"))
+ wd/"include"
+ else
+ wd/"daffodil-runtime2"/"src"/"main"/"c"
+ val libDir = if (os.exists(wd/"lib"))
+ wd/"lib"
+ else
+
wd/"daffodil-runtime2"/"target"/"streams"/"compile"/"ccTargetMap"/"_global"/"streams"/"compile"/"sbtcc.Library"
Review comment:
The logic indeed is overly complicated here in order to allow me to run
a test from the IDE (in which case we need to look for the headers and static
lib in sbt paths) as well as run a TDML test from the daffodil command line (in
which case we need to look for the headers and static lib in
daffodil/{include,lib}). If we don't want to check sbt paths in the logic
here, then we need to extract the headers and static lib (or headers and source
files) from our jar at runtime. In the latter case, we want to cache the
static lib somewhere to avoid compiling it every time this method is called.
##########
File path: daffodil-cli/build.sbt
##########
@@ -42,6 +42,18 @@ mappings in Universal ++= Seq(
baseDirectory.value / "README.md" -> "README.md",
)
+mappings in Universal ++= Seq(
+ baseDirectory.value / ".." / "daffodil-runtime2" / "src" / "main" / "c" /
"common_runtime.h" -> "include/common_runtime.h",
+ baseDirectory.value / ".." / "daffodil-runtime2" / "src" / "main" / "c" /
"daffodil_argp.h" -> "include/daffodil_argp.h",
+ baseDirectory.value / ".." / "daffodil-runtime2" / "src" / "main" / "c" /
"stack.h" -> "include/stack.h",
+ baseDirectory.value / ".." / "daffodil-runtime2" / "src" / "main" / "c" /
"xml_reader.h" -> "include/xml_reader.h",
+ baseDirectory.value / ".." / "daffodil-runtime2" / "src" / "main" / "c" /
"xml_writer.h" -> "include/xml_writer.h",
+)
+
+mappings in Universal ++= Seq(
+ baseDirectory.value / ".." / "daffodil-runtime2" / "target" / "streams" /
"compile" / "ccTargetMap" / "_global" / "streams" / "compile" / "sbtcc.Library"
/ "libruntime2.a" -> "lib/libruntime2.a",
Review comment:
Yes, the cc plugin does make a property available to get access to the
static lib. I was able to use that property when I had these mappings in
Universal inside the daffodil-runtime2 project in the parent directory's
build.sbt, but I wasn't able to figure out how to use that property when I
moved the mappings to daffodil-cli/build.sbt here. If anyone can look at
<https://github.com/tnakamot/sbt-cc/blob/master/examples/packaging-with-universal/README.md>
and figure out a way, I'd appreciate it.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/runtime2/generators/ParserGenerator.scala
##########
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2.generators
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.dpath.NodeInfo
+import org.apache.daffodil.dpath.NodeInfo.PrimType
+import org.apache.daffodil.dsom.ElementBase
+import org.apache.daffodil.exceptions.ThrowsSDE
+
+import scala.collection.mutable
+
+/**
+ * Gives an object the ability to generate code.
+ */
+trait ParserGenerator {
+ def generateCode(state: CodeGeneratorState): Unit
+}
+
+/**
+ * Builds up the state of generated code.
+ */
+class CodeGeneratorState extends DFDL.CodeGeneratorState {
+ private val structs = mutable.Stack[ComplexCGState]()
+ private val prototypes = mutable.ArrayBuffer[String]()
+ private val erds = mutable.ArrayBuffer[String]()
+ private val finalStructs = mutable.ArrayBuffer[String]()
+ private val finalImplementation = mutable.ArrayBuffer[String]()
+
+ def addImplementation(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ val initStatements = structs.top.initStatements.mkString("\n")
+ val parserStatements = structs.top.parserStatements.mkString("\n")
+ val unparserStatements = structs.top.unparserStatements.mkString("\n")
+ val prototypeFunctions =
+ s"""static void ${C}_initSelf($C *instance);
+ |static const char *${C}_parseSelf($C *instance, const PState
*pstate);
+ |static const char *${C}_unparseSelf(const $C *instance, const UState
*ustate);""".stripMargin
+ prototypes += prototypeFunctions
+ val functions =
+ s"""static void
+ |${C}_initSelf($C *instance)
+ |{
+ |$initStatements
+ |}
+ |
+ |static const char *
+ |${C}_parseSelf($C *instance, const PState *pstate)
+ |{
+ | const char *error_msg = NULL;
+ |$parserStatements
+ | return error_msg;
+ |}
+ |
+ |static const char *
+ |${C}_unparseSelf(const $C *instance, const UState *ustate)
+ |{
+ | const char *error_msg = NULL;
+ |$unparserStatements
+ | return error_msg;
+ |}
+ |""".stripMargin
+ finalImplementation += functions
+ }
+
+ private def defineQNameInit(context: ElementBase): String = {
+ val qname = context.namedQName.toQNameString
+ val xmlns = if (context.namedQName.prefix.isDefined)
s"xmlns:${context.namedQName.prefix.get}" else "xmlns"
+ val ns = context.namedQName.namespace.toStringOrNullIfNoNS
+ // Optimize away xmlns=ns declaration if possible, although this approach
may not be entirely correct
+ val parentOpt = context.enclosingElements.headOption
+ val parentNs = if (parentOpt.isDefined)
parentOpt.get.namedQName.namespace.toStringOrNullIfNoNS
+ val qnameInit = if (ns == null || ns == parentNs)
+ s""" {"$qname"}, // namedQName.name"""
+ else
+ s""" {
+ | "$qname", // namedQName.name
+ | "$xmlns", // namedQName.xmlns
+ | "$ns", // namedQName.ns
+ | },""".stripMargin
+ qnameInit
+ }
+
+ def addComplexTypeERD(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ val count = structs.top.declarations.length
+ val offsetComputations = structs.top.offsetComputations.mkString(",\n")
+ val erdComputations = structs.top.erdComputations.mkString(",\n")
+ val qnameInit = defineQNameInit(context)
+ val complexERD =
+ s"""static const $C ${C}_compute_ERD_offsets;
+ |
+ |static const ptrdiff_t ${C}_offsets[$count] = {
+ |$offsetComputations
+ |};
+ |
+ |static const ERD *${C}_childrenERDs[$count] = {
+ |$erdComputations
+ |};
+ |
+ |static const ERD ${C}_ERD = {
+ |$qnameInit
+ | COMPLEX, // typeCode
+ | $count, // numChildren
+ | ${C}_offsets, // offsets
+ | ${C}_childrenERDs, // childrenERDs
+ | (ERDInitSelf)&${C}_initSelf, // initSelf
+ | (ERDParseSelf)&${C}_parseSelf, // parseSelf
+ | (ERDUnparseSelf)&${C}_unparseSelf, // unparseSelf
+ |};
+ |""".stripMargin
+ erds += complexERD
+ }
+
+ def addStruct(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ val declarations = structs.top.declarations.mkString("\n")
+ val struct =
+ s"""typedef struct $C
+ |{
+ | InfosetBase _base;
+ |$declarations
+ |} $C;
+ |""".stripMargin
+ finalStructs += struct
+ val initStatement = s" instance->_base.erd = &${C}_ERD;"
+ structs.top.initStatements += initStatement
+ }
+
+ def addSimpleTypeStatements(initStatement: String, parseStatement: String,
unparseStatement: String): Unit = {
+ structs.top.initStatements += initStatement
+ structs.top.parserStatements += parseStatement
+ structs.top.unparserStatements += unparseStatement
+ }
+
+ def addComplexTypeStatements(child: ElementBase): Unit = {
+ val C = child.namedQName.local
+ val e = child.name
+ val initStatement = s" ${C}_initSelf(&instance->$e);"
+ val parseStatement =
+ s""" if (error_msg == NULL)
+ | {
+ | error_msg = ${C}_parseSelf(&instance->$e, pstate);
+ | }""".stripMargin
+ val unparseStatement =
+ s""" if (error_msg == NULL)
+ | {
+ | error_msg = ${C}_unparseSelf(&instance->$e, ustate);
+ | }""".stripMargin
+ structs.top.initStatements += initStatement
+ structs.top.parserStatements += parseStatement
+ structs.top.unparserStatements += unparseStatement
+ }
+
+ def pushComplexElement(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ structs.push(new ComplexCGState(C))
+ }
+
+ def popComplexElement(context: ElementBase): Unit = {
+ structs.pop()
+ }
+
+ def addSimpleTypeERD(context: ElementBase): Unit = {
+ val e = context.namedQName.local
+ val qnameInit = defineQNameInit(context)
+ val typeCode = context.optPrimType.get match {
+ case PrimType.Int => "PRIMITIVE_INT32"
+ case PrimType.String => "PRIMITIVE_STRING"
+ case p: PrimType => context.SDE("PrimType %s not supported yet.",
p.toString)
+ }
+ val erd =
+ s"""static const ERD ${e}_ERD = {
+ |$qnameInit
+ | $typeCode, // typeCode
+ | 0, // numChildren
+ | NULL, // offsets
+ | NULL, // childrenERDs
+ | NULL, // initSelf
+ | NULL, // parseSelf
+ | NULL, // unparseSelf
+ |};
+ |""".stripMargin
+ erds += erd
+ addComputations(context)
+ }
+
+ def addComputations(child: ElementBase): Unit = {
+ val C = structs.top.C
+ val e = child.namedQName.local
+ val offsetComputation = s" (char *)&${C}_compute_ERD_offsets.$e - (char
*)&${C}_compute_ERD_offsets"
+ val erdComputation = s" &${e}_ERD"
+ structs.top.offsetComputations += offsetComputation
+ structs.top.erdComputations += erdComputation
+ }
+
+ def addFieldDeclaration(context: ThrowsSDE, child: ElementBase): Unit = {
+ val definition = if (child.isSimpleType) {
+ import NodeInfo.PrimType
+ child.optPrimType.get match {
+ case PrimType.Long => "int64_t "
+ case PrimType.Int => "int32_t "
+ case x => context.SDE("Unsupported primitive type: " + x)
+ }
+ } else {
+ child.namedQName.local + " "
+ }
+ structs.top.declarations += s" $definition ${child.name};"
+ }
+
+ def viewCodeHeader: String = {
Review comment:
These members were named with "view" prefix due to an old method named
"view" which returned a string of C code passed into the object. Yes,
"generate" would be a better prefix.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/runtime2/generators/ParserGenerator.scala
##########
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2.generators
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.dpath.NodeInfo
+import org.apache.daffodil.dpath.NodeInfo.PrimType
+import org.apache.daffodil.dsom.ElementBase
+import org.apache.daffodil.exceptions.ThrowsSDE
+
+import scala.collection.mutable
+
+/**
+ * Gives an object the ability to generate code.
+ */
+trait ParserGenerator {
+ def generateCode(state: CodeGeneratorState): Unit
+}
+
+/**
+ * Builds up the state of generated code.
+ */
+class CodeGeneratorState extends DFDL.CodeGeneratorState {
+ private val structs = mutable.Stack[ComplexCGState]()
+ private val prototypes = mutable.ArrayBuffer[String]()
+ private val erds = mutable.ArrayBuffer[String]()
+ private val finalStructs = mutable.ArrayBuffer[String]()
+ private val finalImplementation = mutable.ArrayBuffer[String]()
+
+ def addImplementation(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ val initStatements = structs.top.initStatements.mkString("\n")
+ val parserStatements = structs.top.parserStatements.mkString("\n")
+ val unparserStatements = structs.top.unparserStatements.mkString("\n")
+ val prototypeFunctions =
+ s"""static void ${C}_initSelf($C *instance);
+ |static const char *${C}_parseSelf($C *instance, const PState
*pstate);
+ |static const char *${C}_unparseSelf(const $C *instance, const UState
*ustate);""".stripMargin
+ prototypes += prototypeFunctions
+ val functions =
+ s"""static void
+ |${C}_initSelf($C *instance)
+ |{
+ |$initStatements
+ |}
+ |
+ |static const char *
+ |${C}_parseSelf($C *instance, const PState *pstate)
+ |{
+ | const char *error_msg = NULL;
+ |$parserStatements
+ | return error_msg;
+ |}
+ |
+ |static const char *
+ |${C}_unparseSelf(const $C *instance, const UState *ustate)
+ |{
+ | const char *error_msg = NULL;
+ |$unparserStatements
+ | return error_msg;
+ |}
+ |""".stripMargin
+ finalImplementation += functions
+ }
+
+ private def defineQNameInit(context: ElementBase): String = {
+ val qname = context.namedQName.toQNameString
+ val xmlns = if (context.namedQName.prefix.isDefined)
s"xmlns:${context.namedQName.prefix.get}" else "xmlns"
+ val ns = context.namedQName.namespace.toStringOrNullIfNoNS
+ // Optimize away xmlns=ns declaration if possible, although this approach
may not be entirely correct
+ val parentOpt = context.enclosingElements.headOption
+ val parentNs = if (parentOpt.isDefined)
parentOpt.get.namedQName.namespace.toStringOrNullIfNoNS
+ val qnameInit = if (ns == null || ns == parentNs)
Review comment:
Agreed, will avoid null pointers by using Option types and map { => }.
##########
File path:
daffodil-core/src/test/scala/org/apache/daffodil/runtime2/TestGeneratedCodeCompiler.scala
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import java.io.ByteArrayInputStream
+
+import org.apache.daffodil.compiler.Compiler
+import org.apache.daffodil.infoset.DIComplex
+import org.apache.daffodil.infoset.TestInfoset
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.util.SchemaUtils
+import org.junit.Test
+
+// Now that we can run TDML tests with runtime2, this test's remaining
+// value is for debugging of runtime2 components.
+class TestGeneratedCodeCompiler {
+
+ @Test
+ def compileRunParseInt32(): Unit = {
+ // Compile a DFDL schema to parse int32 numbers
+ val testSchema = SchemaUtils.dfdlTestSchema(
+ <xs:include
schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>,
+ <dfdl:format representation="binary" ref="GeneralFormat"/>,
+ <xs:element name="c1">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="e1" type="xs:int"/>
+ <xs:element name="c2">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="e2" type="xs:int"/>
Review comment:
We probably need a sequence containing a repeated element with minOccurs
> 1.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/grammar/primitives/SequenceCombinator.scala
##########
@@ -91,6 +92,16 @@ class OrderedSequence(sq: SequenceTermBase,
sequenceChildrenArg: Seq[SequenceChi
}
}
}
+
+ override def generateCode(cgState: CodeGeneratorState): Unit = {
+ //
+ // To lift this draconian restriction, we have to
+ // generate code for each of the children, and combine them into a block
+ //
+ sq.schemaDefinitionUnless(sequenceChildren.length == 1, "Only a single
child of a sequence is supported.")
Review comment:
Hmm, our test does have a nested complex struct type with two elements
both of which are 32-bit ints. I can only think that Daffodil must have turned
that complex type into something different than an OrderedSequence with two
children. Maybe it was because each element had its own definition and name
(simple types e2 and e3) and we needed a simple type e2 in a sequence with
minOccurs = 2 to create an OrderedSequence with two children.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/runtime2/generators/ParserGenerator.scala
##########
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2.generators
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.dpath.NodeInfo
+import org.apache.daffodil.dpath.NodeInfo.PrimType
+import org.apache.daffodil.dsom.ElementBase
+import org.apache.daffodil.exceptions.ThrowsSDE
+
+import scala.collection.mutable
+
+/**
+ * Gives an object the ability to generate code.
+ */
+trait ParserGenerator {
+ def generateCode(state: CodeGeneratorState): Unit
+}
+
+/**
+ * Builds up the state of generated code.
+ */
+class CodeGeneratorState extends DFDL.CodeGeneratorState {
+ private val structs = mutable.Stack[ComplexCGState]()
+ private val prototypes = mutable.ArrayBuffer[String]()
+ private val erds = mutable.ArrayBuffer[String]()
+ private val finalStructs = mutable.ArrayBuffer[String]()
+ private val finalImplementation = mutable.ArrayBuffer[String]()
+
+ def addImplementation(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ val initStatements = structs.top.initStatements.mkString("\n")
+ val parserStatements = structs.top.parserStatements.mkString("\n")
+ val unparserStatements = structs.top.unparserStatements.mkString("\n")
+ val prototypeFunctions =
+ s"""static void ${C}_initSelf($C *instance);
+ |static const char *${C}_parseSelf($C *instance, const PState
*pstate);
+ |static const char *${C}_unparseSelf(const $C *instance, const UState
*ustate);""".stripMargin
+ prototypes += prototypeFunctions
+ val functions =
+ s"""static void
+ |${C}_initSelf($C *instance)
+ |{
+ |$initStatements
+ |}
+ |
+ |static const char *
+ |${C}_parseSelf($C *instance, const PState *pstate)
+ |{
+ | const char *error_msg = NULL;
+ |$parserStatements
+ | return error_msg;
+ |}
+ |
+ |static const char *
+ |${C}_unparseSelf(const $C *instance, const UState *ustate)
+ |{
+ | const char *error_msg = NULL;
+ |$unparserStatements
+ | return error_msg;
+ |}
+ |""".stripMargin
+ finalImplementation += functions
+ }
+
+ private def defineQNameInit(context: ElementBase): String = {
+ val qname = context.namedQName.toQNameString
+ val xmlns = if (context.namedQName.prefix.isDefined)
s"xmlns:${context.namedQName.prefix.get}" else "xmlns"
+ val ns = context.namedQName.namespace.toStringOrNullIfNoNS
+ // Optimize away xmlns=ns declaration if possible, although this approach
may not be entirely correct
+ val parentOpt = context.enclosingElements.headOption
+ val parentNs = if (parentOpt.isDefined)
parentOpt.get.namedQName.namespace.toStringOrNullIfNoNS
+ val qnameInit = if (ns == null || ns == parentNs)
+ s""" {"$qname"}, // namedQName.name"""
+ else
+ s""" {
+ | "$qname", // namedQName.name
+ | "$xmlns", // namedQName.xmlns
Review comment:
We are initializing static variables, which ensures missing members are
initialized to null. We can be more explicit and use complete initializers
instead of short initializers if necessary.
##########
File path:
daffodil-core/src/test/scala/org/apache/daffodil/runtime2/TestGeneratedCodeCompiler.scala
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import java.io.ByteArrayInputStream
+
+import org.apache.daffodil.compiler.Compiler
+import org.apache.daffodil.infoset.DIComplex
+import org.apache.daffodil.infoset.TestInfoset
+import org.apache.daffodil.util.Misc
+import org.apache.daffodil.util.SchemaUtils
+import org.junit.Test
+
+// Now that we can run TDML tests with runtime2, this test's remaining
+// value is for debugging of runtime2 components.
+class TestGeneratedCodeCompiler {
+
+ @Test
+ def compileRunParseInt32(): Unit = {
+ // Compile a DFDL schema to parse int32 numbers
+ val testSchema = SchemaUtils.dfdlTestSchema(
+ <xs:include
schemaLocation="org/apache/daffodil/xsd/DFDLGeneralFormat.dfdl.xsd"/>,
+ <dfdl:format representation="binary" ref="GeneralFormat"/>,
+ <xs:element name="c1">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="e1" type="xs:int"/>
+ <xs:element name="c2">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="e2" type="xs:int"/>
+ <xs:element name="e3" type="xs:int"/>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>)
+ val schemaCompiler = Compiler()
+ val pf = schemaCompiler.compileNode(testSchema)
+ assert(!pf.isError, pf.getDiagnostics.map(_.getMessage()).mkString("\n"))
+ // Generate C code from the DFDL schema
+ val codeGeneratorState = pf.generateCode()
+ val generatedCodeCompiler = new GeneratedCodeCompiler(pf)
+ val rootElementName = "c1"
+ generatedCodeCompiler.compile(rootElementName, codeGeneratorState)
+ assert(!pf.isError, pf.getDiagnostics.map(_.getMessage()).mkString("\n"))
+ // Run the executable to parse int32 numbers
+ val dp = generatedCodeCompiler.dataProcessor
+ val b = Misc.hex2Bytes("000000010000000200000003")
+ val input = new ByteArrayInputStream(b)
+ val pr = dp.parse(input)
+ assert(!pr.isError && pf.getDiagnostics.isEmpty,
pr.getDiagnostics.map(_.getMessage()).mkString("\n"))
+ // Create an internal Daffodil infoset from the XML file
+ val (infoset: DIComplex, _, tunables) =
TestInfoset.testInfoset(testSchema, pr.infosetAsXML)
+ assert(infoset.hasVisibleChildren)
+ assert(infoset.erd.name == "c1")
Review comment:
Good idea, I would like a way to validate the infoset against the schema.
##########
File path: daffodil-runtime2/src/main/c/daffodil_main.c
##########
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "common_runtime.h" // for walkInfoset, InfosetBase, ERD, ...
+#include "daffodil_argp.h" // for daffodil_cli, parse_daffodil_cli, ...
+#include "generated_code.h" // for rootInfoset
Review comment:
I put the prototype for the rootInfoset() function in generate_code.h
because generate_code.c implements that function. The prototype is generic,
however, so it could be in another header file or inlined here.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/runtime2/generators/ParserGenerator.scala
##########
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2.generators
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.dpath.NodeInfo
+import org.apache.daffodil.dpath.NodeInfo.PrimType
+import org.apache.daffodil.dsom.ElementBase
+import org.apache.daffodil.exceptions.ThrowsSDE
+
+import scala.collection.mutable
+
+/**
+ * Gives an object the ability to generate code.
+ */
+trait ParserGenerator {
+ def generateCode(state: CodeGeneratorState): Unit
+}
+
+/**
+ * Builds up the state of generated code.
+ */
+class CodeGeneratorState extends DFDL.CodeGeneratorState {
+ private val structs = mutable.Stack[ComplexCGState]()
+ private val prototypes = mutable.ArrayBuffer[String]()
+ private val erds = mutable.ArrayBuffer[String]()
+ private val finalStructs = mutable.ArrayBuffer[String]()
+ private val finalImplementation = mutable.ArrayBuffer[String]()
+
+ def addImplementation(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ val initStatements = structs.top.initStatements.mkString("\n")
+ val parserStatements = structs.top.parserStatements.mkString("\n")
+ val unparserStatements = structs.top.unparserStatements.mkString("\n")
+ val prototypeFunctions =
+ s"""static void ${C}_initSelf($C *instance);
+ |static const char *${C}_parseSelf($C *instance, const PState
*pstate);
+ |static const char *${C}_unparseSelf(const $C *instance, const UState
*ustate);""".stripMargin
+ prototypes += prototypeFunctions
+ val functions =
+ s"""static void
+ |${C}_initSelf($C *instance)
+ |{
+ |$initStatements
+ |}
+ |
+ |static const char *
+ |${C}_parseSelf($C *instance, const PState *pstate)
+ |{
+ | const char *error_msg = NULL;
+ |$parserStatements
+ | return error_msg;
+ |}
+ |
+ |static const char *
+ |${C}_unparseSelf(const $C *instance, const UState *ustate)
+ |{
+ | const char *error_msg = NULL;
+ |$unparserStatements
+ | return error_msg;
+ |}
+ |""".stripMargin
+ finalImplementation += functions
+ }
+
+ private def defineQNameInit(context: ElementBase): String = {
+ val qname = context.namedQName.toQNameString
+ val xmlns = if (context.namedQName.prefix.isDefined)
s"xmlns:${context.namedQName.prefix.get}" else "xmlns"
+ val ns = context.namedQName.namespace.toStringOrNullIfNoNS
+ // Optimize away xmlns=ns declaration if possible, although this approach
may not be entirely correct
+ val parentOpt = context.enclosingElements.headOption
+ val parentNs = if (parentOpt.isDefined)
parentOpt.get.namedQName.namespace.toStringOrNullIfNoNS
+ val qnameInit = if (ns == null || ns == parentNs)
+ s""" {"$qname"}, // namedQName.name"""
+ else
+ s""" {
+ | "$qname", // namedQName.name
+ | "$xmlns", // namedQName.xmlns
+ | "$ns", // namedQName.ns
+ | },""".stripMargin
+ qnameInit
+ }
+
+ def addComplexTypeERD(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ val count = structs.top.declarations.length
+ val offsetComputations = structs.top.offsetComputations.mkString(",\n")
+ val erdComputations = structs.top.erdComputations.mkString(",\n")
+ val qnameInit = defineQNameInit(context)
+ val complexERD =
+ s"""static const $C ${C}_compute_ERD_offsets;
+ |
+ |static const ptrdiff_t ${C}_offsets[$count] = {
+ |$offsetComputations
+ |};
+ |
+ |static const ERD *${C}_childrenERDs[$count] = {
+ |$erdComputations
+ |};
+ |
+ |static const ERD ${C}_ERD = {
+ |$qnameInit
+ | COMPLEX, // typeCode
+ | $count, // numChildren
+ | ${C}_offsets, // offsets
+ | ${C}_childrenERDs, // childrenERDs
+ | (ERDInitSelf)&${C}_initSelf, // initSelf
+ | (ERDParseSelf)&${C}_parseSelf, // parseSelf
+ | (ERDUnparseSelf)&${C}_unparseSelf, // unparseSelf
+ |};
+ |""".stripMargin
+ erds += complexERD
+ }
+
+ def addStruct(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ val declarations = structs.top.declarations.mkString("\n")
+ val struct =
+ s"""typedef struct $C
+ |{
+ | InfosetBase _base;
+ |$declarations
+ |} $C;
+ |""".stripMargin
+ finalStructs += struct
+ val initStatement = s" instance->_base.erd = &${C}_ERD;"
+ structs.top.initStatements += initStatement
+ }
+
+ def addSimpleTypeStatements(initStatement: String, parseStatement: String,
unparseStatement: String): Unit = {
+ structs.top.initStatements += initStatement
+ structs.top.parserStatements += parseStatement
+ structs.top.unparserStatements += unparseStatement
+ }
+
+ def addComplexTypeStatements(child: ElementBase): Unit = {
+ val C = child.namedQName.local
+ val e = child.name
+ val initStatement = s" ${C}_initSelf(&instance->$e);"
+ val parseStatement =
+ s""" if (error_msg == NULL)
+ | {
+ | error_msg = ${C}_parseSelf(&instance->$e, pstate);
+ | }""".stripMargin
+ val unparseStatement =
+ s""" if (error_msg == NULL)
+ | {
+ | error_msg = ${C}_unparseSelf(&instance->$e, ustate);
+ | }""".stripMargin
+ structs.top.initStatements += initStatement
+ structs.top.parserStatements += parseStatement
+ structs.top.unparserStatements += unparseStatement
+ }
+
+ def pushComplexElement(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ structs.push(new ComplexCGState(C))
+ }
+
+ def popComplexElement(context: ElementBase): Unit = {
+ structs.pop()
+ }
+
+ def addSimpleTypeERD(context: ElementBase): Unit = {
+ val e = context.namedQName.local
+ val qnameInit = defineQNameInit(context)
+ val typeCode = context.optPrimType.get match {
+ case PrimType.Int => "PRIMITIVE_INT32"
+ case PrimType.String => "PRIMITIVE_STRING"
+ case p: PrimType => context.SDE("PrimType %s not supported yet.",
p.toString)
+ }
+ val erd =
+ s"""static const ERD ${e}_ERD = {
+ |$qnameInit
+ | $typeCode, // typeCode
+ | 0, // numChildren
+ | NULL, // offsets
+ | NULL, // childrenERDs
+ | NULL, // initSelf
+ | NULL, // parseSelf
+ | NULL, // unparseSelf
+ |};
+ |""".stripMargin
+ erds += erd
+ addComputations(context)
+ }
+
+ def addComputations(child: ElementBase): Unit = {
+ val C = structs.top.C
+ val e = child.namedQName.local
+ val offsetComputation = s" (char *)&${C}_compute_ERD_offsets.$e - (char
*)&${C}_compute_ERD_offsets"
+ val erdComputation = s" &${e}_ERD"
+ structs.top.offsetComputations += offsetComputation
+ structs.top.erdComputations += erdComputation
+ }
+
+ def addFieldDeclaration(context: ThrowsSDE, child: ElementBase): Unit = {
+ val definition = if (child.isSimpleType) {
+ import NodeInfo.PrimType
+ child.optPrimType.get match {
+ case PrimType.Long => "int64_t "
+ case PrimType.Int => "int32_t "
+ case x => context.SDE("Unsupported primitive type: " + x)
+ }
+ } else {
+ child.namedQName.local + " "
+ }
+ structs.top.declarations += s" $definition ${child.name};"
+ }
+
+ def viewCodeHeader: String = {
+ val structs = finalStructs.mkString("\n")
+ val header =
+ s"""#ifndef GENERATED_CODE_H
+ |#define GENERATED_CODE_H
+ |
+ |#include "common_runtime.h" // for InfosetBase
+ |#include <stdint.h> // for int32_t
+ |
+ |// Return the root of an infoset to be used for parsing or unparsing
+ |
+ |extern InfosetBase *rootInfoset();
+ |
+ |// Define some infoset structures
+ |
+ |$structs
+ |#endif // GENERATED_CODE_H
+ |""".stripMargin
+ header
+ }
+
+ def viewCodeFile(rootElementName: String): String = {
+ val prototypes = this.prototypes.mkString("\n")
+ val erds = this.erds.mkString("\n")
+ val finalImplementation = this.finalImplementation.mkString("\n")
+ val code =
+ s"""#include "generated_code.h"
+ |#include <endian.h> // for be32toh, htobe32
+ |#include <errno.h> // for errno
+ |#include <stddef.h> // for ptrdiff_t
+ |#include <stdio.h> // for NULL, fread, fwrite, size_t, feof,
ferror, FILE
+ |#include <string.h> // for strerror
+ |
+ |// Prototypes needed for compilation
+ |
+ |$prototypes
+ |
+ |// Metadata singletons
+ |
+ |$erds
+ |// Return the root of an infoset to be used for parsing or unparsing
+ |
+ |InfosetBase *
+ |rootInfoset()
+ |{
+ | static $rootElementName instance;
+ | InfosetBase *root = &instance._base;
+ | ${rootElementName}_ERD.initSelf(root);
+ | return root;
+ |}
+ |
+ |// Methods to initialize, parse, and unparse infoset nodes
+ |
+ |static const char *
Review comment:
Yes, eof_or_error_msg can be static code in libruntime2.a; I simply
hadn't considered the possibility of it being defined outside of the
generate_code.c file.
##########
File path:
daffodil-runtime1/src/main/scala/org/apache/daffodil/api/DFDLParserUnparser.scala
##########
@@ -202,6 +204,14 @@ object DFDL {
def parse(ab: Array[Byte]): Unit
}
+ trait CodeGeneratorState {
Review comment:
I also wasn't sure if we needed this either. I put my best guess in a
comment below this line why I thought it was there.
##########
File path: daffodil-runtime2/src/main/c/common_runtime.c
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "common_runtime.h"
+
+// walkInfosetNode - recursively walk an infoset node and call
+// VisitEventHandler methods
+
+static const char *
+walkInfosetNode(const VisitEventHandler *handler, const InfosetBase *infoNode)
Review comment:
Yes, I know sphinx and other similar tools exist. Will look into using
one of them.
##########
File path: daffodil-runtime2/src/main/c/common_runtime.h
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef COMMON_RUNTIME_H
+#define COMMON_RUNTIME_H
+
+#include <stddef.h> // for ptrdiff_t
+#include <stdint.h> // for int32_t
+#include <stdio.h> // for FILE, size_t
+
+// Prototypes needed for compilation
+
+typedef struct ElementRuntimeData ERD;
+typedef struct InfosetBase InfosetBase;
+typedef struct PState PState;
+typedef struct UState UState;
+typedef struct VisitEventHandler VisitEventHandler;
+
+typedef void (*ERDInitSelf)(InfosetBase *infoNode);
+typedef const char *(*ERDParseSelf)(InfosetBase * infoNode,
+ const PState *pstate);
+typedef const char *(*ERDUnparseSelf)(const InfosetBase *infoNode,
+ const UState * ustate);
+
+typedef const char *(*VisitStartDocument)(const VisitEventHandler *handler);
+typedef const char *(*VisitEndDocument)(const VisitEventHandler *handler);
+typedef const char *(*VisitStartComplex)(const VisitEventHandler *handler,
+ const InfosetBase * base);
+typedef const char *(*VisitEndComplex)(const VisitEventHandler *handler,
+ const InfosetBase * base);
+typedef const char *(*VisitInt32Elem)(const VisitEventHandler *handler,
+ const ERD *erd, const int32_t *location);
+
+// NamedQName - name of an infoset element
+
+typedef struct NamedQName
+{
+ char *name; // element name (including prefix if any)
+ char *xmlns; // xmlns attribute name (including prefix if any)
Review comment:
Yes, this part of the ERD was TDML runner and XML-conversion specific.
I'm fine with pushing back and asking to keep things needed only for TDML/XML
separate.
##########
File path: daffodil-runtime2/src/main/c/xml_reader.c
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "xml_reader.h"
+#include <errno.h> // for errno, ERANGE
+#include <limits.h> // for LONG_MAX, LONG_MIN
+#include <mxml.h> // for mxmlWalkNext, mxmlGetElement, mxmlGetType, ...
+#include <stdint.h> // for int32_t, INT32_MAX, INT32_MIN
+#include <stdlib.h> // for NULL, strtol
+#include <string.h> // for strcmp, strerror
+
+// Read XML data from file before walking infoset
+
+static const char *
+xmlStartDocument(XMLReader *reader)
+{
+ // Load the XML data into memory
+ reader->xml = mxmlLoadFile(NULL, reader->stream, MXML_OPAQUE_CALLBACK);
+ reader->node = reader->xml;
+ if (reader->node == NULL) {
+ return "Unable to read XML data from input file";
+ }
+
+ // Consume the <?xml line if there is one
+ const char *name = mxmlGetElement(reader->node);
+ if (name && strncmp(name, "?xml", strlen("?xml")) == 0) {
+ do
+ {
+ reader->node = mxmlWalkNext(reader->node, reader->xml,
MXML_DESCEND);
+ } while (mxmlGetType(reader->node) == MXML_OPAQUE);
+ name = mxmlGetElement(reader->node);
+ }
+
+ // Consume a comment if there is one
+ if (name && strncmp(name, "!--", strlen("!--")) == 0) {
+ do
+ {
+ reader->node = mxmlWalkNext(reader->node, reader->xml,
MXML_DESCEND);
+ } while (mxmlGetType(reader->node) == MXML_OPAQUE);
+ }
+
+ return reader->node ? NULL : "Ran out of XML data";
+}
+
+// Delete XML data after walking infoset
+
+static const char *
+xmlEndDocument(XMLReader *reader)
+{
+ // Consume any remaining newlines or whitespace
+ while (mxmlGetType(reader->node) == MXML_OPAQUE) {
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+ }
+
+ // Check whether we have consumed all of the XML data
+ if (reader->node != NULL) {
+ // This code path exits the program - no need to call mxmlDelete
+ return "Did not consume all of the XML data";
+ }
+
+ // Free the storage allocated to hold the XML data
+ mxmlDelete(reader->xml);
+ reader->xml = NULL;
+ reader->node = NULL;
+ return NULL;
+}
+
+// Continue walking both XML data and infoset in lockstep
+
+static const char *
+xmlStartComplex(XMLReader *reader, const InfosetBase *base)
+{
+ // Consume any newlines or whitespace before the element
+ while (mxmlGetType(reader->node) == MXML_OPAQUE) {
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+ }
+
+ // Get the element and consume it
+ const char *name_from_xml = mxmlGetElement(reader->node);
+ const char *name_from_infoset = base->erd->namedQName.name;
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+
+ // Check whether we are walking both XML data and infoset in lockstep
+ if (name_from_xml && name_from_infoset)
+ {
+ return strcmp(name_from_xml, name_from_infoset) == 0
+ ? NULL
+ : "Found mismatch between XML data and infoset";
+ }
+ else
+ {
+ return "Ran out of XML data";
+ }
+}
+
+// Consume XML data only on start events, not end events
+
+static const char *
+xmlEndComplex(XMLReader *reader, const InfosetBase *base)
+{
+ (void)reader;
+ (void)base;
+ return NULL;
+}
+
+// Read 32-bit integer value from XML data
+
+static const char *
+xmlInt32Elem(XMLReader *reader, const ERD *erd, int32_t *location)
+{
+ // Consume any newlines or whitespace before the element
+ while (mxmlGetType(reader->node) == MXML_OPAQUE) {
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+ }
+
+ // Get the element and consume it
+ const char *name_from_xml = mxmlGetElement(reader->node);
+ const char *number_from_xml = mxmlGetOpaque(reader->node);
+ const char *name_from_infoset = erd->namedQName.name;
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
Review comment:
Nope, unless you want to change other places besides this place.
##########
File path: daffodil-runtime2/src/main/c/xml_reader.c
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "xml_reader.h"
+#include <errno.h> // for errno, ERANGE
+#include <limits.h> // for LONG_MAX, LONG_MIN
+#include <mxml.h> // for mxmlWalkNext, mxmlGetElement, mxmlGetType, ...
+#include <stdint.h> // for int32_t, INT32_MAX, INT32_MIN
+#include <stdlib.h> // for NULL, strtol
+#include <string.h> // for strcmp, strerror
+
+// Read XML data from file before walking infoset
+
+static const char *
+xmlStartDocument(XMLReader *reader)
+{
+ // Load the XML data into memory
+ reader->xml = mxmlLoadFile(NULL, reader->stream, MXML_OPAQUE_CALLBACK);
+ reader->node = reader->xml;
+ if (reader->node == NULL) {
+ return "Unable to read XML data from input file";
+ }
+
+ // Consume the <?xml line if there is one
+ const char *name = mxmlGetElement(reader->node);
+ if (name && strncmp(name, "?xml", strlen("?xml")) == 0) {
+ do
+ {
+ reader->node = mxmlWalkNext(reader->node, reader->xml,
MXML_DESCEND);
+ } while (mxmlGetType(reader->node) == MXML_OPAQUE);
+ name = mxmlGetElement(reader->node);
+ }
+
+ // Consume a comment if there is one
+ if (name && strncmp(name, "!--", strlen("!--")) == 0) {
+ do
+ {
+ reader->node = mxmlWalkNext(reader->node, reader->xml,
MXML_DESCEND);
+ } while (mxmlGetType(reader->node) == MXML_OPAQUE);
+ }
+
+ return reader->node ? NULL : "Ran out of XML data";
+}
+
+// Delete XML data after walking infoset
+
+static const char *
+xmlEndDocument(XMLReader *reader)
+{
+ // Consume any remaining newlines or whitespace
+ while (mxmlGetType(reader->node) == MXML_OPAQUE) {
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+ }
+
+ // Check whether we have consumed all of the XML data
+ if (reader->node != NULL) {
+ // This code path exits the program - no need to call mxmlDelete
+ return "Did not consume all of the XML data";
+ }
+
+ // Free the storage allocated to hold the XML data
+ mxmlDelete(reader->xml);
+ reader->xml = NULL;
+ reader->node = NULL;
+ return NULL;
+}
+
+// Continue walking both XML data and infoset in lockstep
+
+static const char *
+xmlStartComplex(XMLReader *reader, const InfosetBase *base)
+{
+ // Consume any newlines or whitespace before the element
+ while (mxmlGetType(reader->node) == MXML_OPAQUE) {
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+ }
+
+ // Get the element and consume it
+ const char *name_from_xml = mxmlGetElement(reader->node);
+ const char *name_from_infoset = base->erd->namedQName.name;
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+
+ // Check whether we are walking both XML data and infoset in lockstep
+ if (name_from_xml && name_from_infoset)
+ {
+ return strcmp(name_from_xml, name_from_infoset) == 0
+ ? NULL
+ : "Found mismatch between XML data and infoset";
+ }
+ else
+ {
+ return "Ran out of XML data";
+ }
+}
+
+// Consume XML data only on start events, not end events
+
+static const char *
+xmlEndComplex(XMLReader *reader, const InfosetBase *base)
+{
+ (void)reader;
+ (void)base;
+ return NULL;
+}
+
+// Read 32-bit integer value from XML data
+
+static const char *
+xmlInt32Elem(XMLReader *reader, const ERD *erd, int32_t *location)
+{
+ // Consume any newlines or whitespace before the element
+ while (mxmlGetType(reader->node) == MXML_OPAQUE) {
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+ }
+
+ // Get the element and consume it
+ const char *name_from_xml = mxmlGetElement(reader->node);
+ const char *number_from_xml = mxmlGetOpaque(reader->node);
+ const char *name_from_infoset = erd->namedQName.name;
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+
+ // Check whether we are walking both XML data and infoset in lockstep
+ if (name_from_xml && name_from_infoset)
+ {
+ if (strcmp(name_from_xml, name_from_infoset) == 0)
+ {
+ // Check for errors when reading the 32-bit integer
+ char * endptr = NULL;
+ errno = 0; // To distinguish success/failure after call
Review comment:
Sounds good, will use strtonum().
##########
File path: daffodil-runtime2/src/main/c/xml_reader.c
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "xml_reader.h"
+#include <errno.h> // for errno, ERANGE
+#include <limits.h> // for LONG_MAX, LONG_MIN
+#include <mxml.h> // for mxmlWalkNext, mxmlGetElement, mxmlGetType, ...
+#include <stdint.h> // for int32_t, INT32_MAX, INT32_MIN
+#include <stdlib.h> // for NULL, strtol
+#include <string.h> // for strcmp, strerror
+
+// Read XML data from file before walking infoset
+
+static const char *
+xmlStartDocument(XMLReader *reader)
+{
+ // Load the XML data into memory
+ reader->xml = mxmlLoadFile(NULL, reader->stream, MXML_OPAQUE_CALLBACK);
+ reader->node = reader->xml;
+ if (reader->node == NULL) {
+ return "Unable to read XML data from input file";
+ }
+
+ // Consume the <?xml line if there is one
+ const char *name = mxmlGetElement(reader->node);
+ if (name && strncmp(name, "?xml", strlen("?xml")) == 0) {
+ do
+ {
+ reader->node = mxmlWalkNext(reader->node, reader->xml,
MXML_DESCEND);
+ } while (mxmlGetType(reader->node) == MXML_OPAQUE);
+ name = mxmlGetElement(reader->node);
+ }
+
+ // Consume a comment if there is one
+ if (name && strncmp(name, "!--", strlen("!--")) == 0) {
+ do
+ {
+ reader->node = mxmlWalkNext(reader->node, reader->xml,
MXML_DESCEND);
+ } while (mxmlGetType(reader->node) == MXML_OPAQUE);
+ }
+
+ return reader->node ? NULL : "Ran out of XML data";
+}
+
+// Delete XML data after walking infoset
+
+static const char *
+xmlEndDocument(XMLReader *reader)
+{
+ // Consume any remaining newlines or whitespace
+ while (mxmlGetType(reader->node) == MXML_OPAQUE) {
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+ }
+
+ // Check whether we have consumed all of the XML data
+ if (reader->node != NULL) {
+ // This code path exits the program - no need to call mxmlDelete
+ return "Did not consume all of the XML data";
+ }
+
+ // Free the storage allocated to hold the XML data
+ mxmlDelete(reader->xml);
+ reader->xml = NULL;
+ reader->node = NULL;
+ return NULL;
+}
+
+// Continue walking both XML data and infoset in lockstep
+
+static const char *
+xmlStartComplex(XMLReader *reader, const InfosetBase *base)
+{
+ // Consume any newlines or whitespace before the element
+ while (mxmlGetType(reader->node) == MXML_OPAQUE) {
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+ }
+
+ // Get the element and consume it
+ const char *name_from_xml = mxmlGetElement(reader->node);
+ const char *name_from_infoset = base->erd->namedQName.name;
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+
+ // Check whether we are walking both XML data and infoset in lockstep
+ if (name_from_xml && name_from_infoset)
+ {
+ return strcmp(name_from_xml, name_from_infoset) == 0
+ ? NULL
+ : "Found mismatch between XML data and infoset";
+ }
+ else
+ {
+ return "Ran out of XML data";
+ }
+}
+
+// Consume XML data only on start events, not end events
+
+static const char *
+xmlEndComplex(XMLReader *reader, const InfosetBase *base)
+{
+ (void)reader;
+ (void)base;
+ return NULL;
+}
+
+// Read 32-bit integer value from XML data
+
+static const char *
+xmlInt32Elem(XMLReader *reader, const ERD *erd, int32_t *location)
+{
+ // Consume any newlines or whitespace before the element
Review comment:
The whitespace being skipped is before the `<foo>43</foo>`, not inside
the `<foo>` element itself.
Not sure if we need to handle elements with CDATA bracketing specially. We
need to test and find out.
Entities are passed through as entities for the application to deal with.
##########
File path:
daffodil-tdml-processor/src/main/scala/org/apache/daffodil/tdml/processor/runtime2/Runtime2TDMLDFDLProcessor.scala
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.tdml.processor.runtime2
+
+import org.apache.daffodil.api._
+import org.apache.daffodil.compiler.Compiler
+import org.apache.daffodil.externalvars.Binding
+import org.apache.daffodil.runtime2.GeneratedCodeCompiler
+import org.apache.daffodil.runtime2.ParseResult
+import org.apache.daffodil.runtime2.UnparseResult
+import org.apache.daffodil.tdml.processor._
+import org.apache.daffodil.xml.XMLUtils
+
+import scala.xml.Node
+
+final class TDMLDFDLProcessorFactory private(
+ private var compiler: Compiler,
+ private var checkAllTopLevel: Boolean,
+ validateDFDLSchemasArg: Boolean)
+ extends AbstractTDMLDFDLProcessorFactory {
+
+ override def validateDFDLSchemas = validateDFDLSchemasArg
+
+ override type R = TDMLDFDLProcessorFactory
+
+ override def implementationName = "daffodil-runtime2"
+
+ def this() = this(compiler = Compiler(validateDFDLSchemas = true),
+ checkAllTopLevel = false,
+ validateDFDLSchemasArg = true)
+
+ private def copy(
+ compiler: Compiler = compiler,
+ checkAllTopLevel: Boolean = checkAllTopLevel,
+ validateDFDLSchemas: Boolean = validateDFDLSchemas) =
+ new TDMLDFDLProcessorFactory(compiler, checkAllTopLevel,
validateDFDLSchemas)
+
+ /**
+ * Deprecated methods must be implemented. Some are just stubs though now.
+ */
+ @deprecated("Use withValidateDFDLSchemas.", "2.6.0")
+ override def setValidateDFDLSchemas(bool: Boolean): Unit = {
+ compiler = compiler.withValidateDFDLSchemas(bool)
+ }
+
+ override def withValidateDFDLSchemas(bool: Boolean):
TDMLDFDLProcessorFactory = {
+ copy(compiler = compiler.withValidateDFDLSchemas(bool))
+ }
+
+ @deprecated("Use withCheckAllTopLevel.", "2.6.0")
+ override def setCheckAllTopLevel(checkAllTopLevel: Boolean): Unit = {
+ compiler = compiler.withCheckAllTopLevel(checkAllTopLevel)
+ }
+
+ override def withCheckAllTopLevel(checkAllTopLevel: Boolean):
TDMLDFDLProcessorFactory = {
+ copy(compiler = compiler.withCheckAllTopLevel(checkAllTopLevel))
+ }
+
+ @deprecated("Use withTunables.", "2.6.0")
+ override def setTunables(tunables: Map[String, String]): Unit =
+ compiler = compiler.withTunables(tunables)
+
+ override def withTunables(tunables: Map[String, String]):
TDMLDFDLProcessorFactory =
+ copy(compiler = compiler.withTunables(tunables))
+
+ @deprecated("Use DaffodilTDMLDFDLProcessor.setExternalDFDLVariables.",
"2.6.0")
+ override def setExternalDFDLVariables(externalVarBindings: Seq[Binding]):
Unit =
+ compiler = compiler.withExternalDFDLVariablesImpl(externalVarBindings)
+
+ override def withExternalDFDLVariables(externalVarBindings: Seq[Binding]):
TDMLDFDLProcessorFactory =
+ copy(compiler =
compiler.withExternalDFDLVariablesImpl(externalVarBindings))
+
+ @deprecated("Use arguments to getProcessor()", "2.6.0")
+ override def setDistinguishedRootNode(name: String, namespace: String): Unit
=
+ compiler = compiler.withDistinguishedRootNode(name, namespace)
+
+ // We're doing to replace this method with different code.
+ // Return result is a TDML.CompileResult - so it's the result
+ // of compiling the schema for the test.
+ override def getProcessor(
+ schemaSource: DaffodilSchemaSource,
+ useSerializedProcessor: Boolean,
+ optRootName: Option[String] = None,
+ optRootNamespace: Option[String] = None): TDML.CompileResult = {
+ val pf = compiler.compileSource(schemaSource, optRootName,
optRootNamespace)
+ val res = if (pf.isError) {
+ Left(pf.getDiagnostics) // DFDL schema compilation diagnostics
+ } else {
+ // How can we move some of these calls to ProcessorFactory with tunable
runtime = "runtime2"?
+ val rootElementName = optRootName.getOrElse("FIXME")
Review comment:
Sure, will add // FIXME as well.
##########
File path: daffodil-runtime2/src/main/c/daffodil_argp.h
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef DAFFODIL_ARGP_H
+#define DAFFODIL_ARGP_H
+
+// Parse our "daffodil" command line interface
+
+extern int parse_daffodil_cli(int argc, char **argv);
+
+// Get our "daffodil" CLI options
+
+extern struct daffodil_cli
+{
+ enum daffodil_subcommand
+ {
+ DAFFODIL_NONE,
+ DAFFODIL_PARSE,
+ DAFFODIL_UNPARSE
+ } subcommand;
+ int verbosity;
+} daffodil_cli;
+
+// Get our "daffodil parse" CLI options
+
+extern struct daffodil_parse_cli
+{
+ const char *infoset_type;
Review comment:
I was mirroring the "--infoset-type" CLI option's name but I'm fine with
"external_infoset_rep".
##########
File path: daffodil-runtime2/src/main/c/.gitignore
##########
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Ignore in case we compile the C files by hand
+*.a
+*.o
+daffodil
+
+# Ignore since these files are not fit for source control
+GPATH
+GRTAGS
+GTAGS
Review comment:
I'll move the excludes to the root .gitignore file.
The G* files are binary files generated by Visual Studio Code to keep track
of where symbols are defined, and they get generated alongside the C source
files by default. I may not be able to change that, but we'll see.
##########
File path: daffodil-runtime2/src/main/c/xml_reader.c
##########
@@ -0,0 +1,186 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "xml_reader.h"
+#include <errno.h> // for errno, ERANGE
+#include <limits.h> // for LONG_MAX, LONG_MIN
+#include <mxml.h> // for mxmlWalkNext, mxmlGetElement, mxmlGetType, ...
+#include <stdint.h> // for int32_t, INT32_MAX, INT32_MIN
+#include <stdlib.h> // for NULL, strtol
+#include <string.h> // for strcmp, strerror
+
+// Read XML data from file before walking infoset
+
+static const char *
+xmlStartDocument(XMLReader *reader)
+{
+ // Load the XML data into memory
+ reader->xml = mxmlLoadFile(NULL, reader->stream, MXML_OPAQUE_CALLBACK);
+ reader->node = reader->xml;
+ if (reader->node == NULL) {
+ return "Unable to read XML data from input file";
+ }
+
+ // Consume the <?xml line if there is one
+ const char *name = mxmlGetElement(reader->node);
+ if (name && strncmp(name, "?xml", strlen("?xml")) == 0) {
+ do
+ {
+ reader->node = mxmlWalkNext(reader->node, reader->xml,
MXML_DESCEND);
+ } while (mxmlGetType(reader->node) == MXML_OPAQUE);
+ name = mxmlGetElement(reader->node);
+ }
+
+ // Consume a comment if there is one
+ if (name && strncmp(name, "!--", strlen("!--")) == 0) {
+ do
+ {
+ reader->node = mxmlWalkNext(reader->node, reader->xml,
MXML_DESCEND);
+ } while (mxmlGetType(reader->node) == MXML_OPAQUE);
+ }
+
+ return reader->node ? NULL : "Ran out of XML data";
+}
+
+// Delete XML data after walking infoset
+
+static const char *
+xmlEndDocument(XMLReader *reader)
+{
+ // Consume any remaining newlines or whitespace
+ while (mxmlGetType(reader->node) == MXML_OPAQUE) {
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+ }
+
+ // Check whether we have consumed all of the XML data
+ if (reader->node != NULL) {
+ // This code path exits the program - no need to call mxmlDelete
+ return "Did not consume all of the XML data";
+ }
+
+ // Free the storage allocated to hold the XML data
+ mxmlDelete(reader->xml);
+ reader->xml = NULL;
+ reader->node = NULL;
+ return NULL;
+}
+
+// Continue walking both XML data and infoset in lockstep
+
+static const char *
+xmlStartComplex(XMLReader *reader, const InfosetBase *base)
+{
+ // Consume any newlines or whitespace before the element
+ while (mxmlGetType(reader->node) == MXML_OPAQUE) {
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+ }
+
+ // Get the element and consume it
+ const char *name_from_xml = mxmlGetElement(reader->node);
+ const char *name_from_infoset = base->erd->namedQName.name;
+ reader->node = mxmlWalkNext(reader->node, reader->xml, MXML_DESCEND);
+
+ // Check whether we are walking both XML data and infoset in lockstep
+ if (name_from_xml && name_from_infoset)
Review comment:
The walkers aren't necessarily aligned with each other until this point.
Some mxmlWalkNext(...) calls above are executed conditionally only as needed
to skip whitespace and then we need the last mxmlWalkNext(...) to skip past the
element's name and set up the next walk.
##########
File path: daffodil-runtime2/src/main/c/generated_code.c
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "generated_code.h"
Review comment:
Correct, this file is an example and it was originally handcrafted. Now
it is identical to what gets generated if given the same schema. However, it
seems reasonable to change its name to generated_example1.c since we might want
to use more examples later.
##########
File path: daffodil-runtime2/src/main/c/daffodil_argp.c
##########
@@ -0,0 +1,302 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "daffodil_argp.h" // for daffodil_cli, daffodil_parse_cli, ...
+#include <argp.h> // for argp_state, argp_error, error_t, argp_parse
+#include <stdio.h> // for sprintf
+#include <stdlib.h> // for putenv, NULL
+#include <string.h> // for strlen, strcmp
+
+// Initialize our "daffodil" name and version
+
+const char *argp_program_version = "Apache Daffodil (runtime2) 0.1";
+
+// Initialize our "daffodil parse" CLI options
+
+struct daffodil_parse_cli daffodil_parse = {
+ "xml", // default infoset type
+ "-", // default infile
+ "-", // default outfile
+};
+
+static const struct argp_option parse_options[] = {
+ {"infoset-type", 'I', "<infoset_type>", 0,
+ "Infoset type to output. Must be one of 'xml' or 'null'"},
+
+ {"output", 'o', "<file>", 0,
+ "Write output to a given file. If not given or is -, output is written to
"
+ "stdout"},
+
+ {0}};
+
+static error_t parse_handler(int key, char *arg, struct argp_state *state);
+
+static const char parse_args_doc[] = "[infile]";
+
+static const char parse_doc[] =
+ "\n"
+ "Parse a file using a DFDL schema\n"
+ "\n"
+ "Parse Options:"
+ "\v"
+ " Trailing arguments:\n"
+ " infile (not required) input file to parse. "
+ "If not specified, or a value of -, reads from stdin";
+
+static const struct argp parse_argp = {
+ parse_options, // array of CLI options
+ parse_handler, // function to get these CLI options
+ parse_args_doc, // short usage documentation
+ parse_doc, // long help documentation
+};
+
+// Handle callbacks to get our "daffodil parse" CLI options
+
+static error_t
+parse_handler(int key, char *arg, struct argp_state *state)
+{
+ struct daffodil_parse_cli *parse = state->input;
+
+ switch (key)
+ {
+ case 'I':
+ parse->infoset_type = arg;
+ break;
+
+ case 'o':
+ parse->outfile = arg;
+ break;
+
+ case ARGP_KEY_ARG:
+ if (state->arg_num)
+ {
+ argp_error(state, "too many arguments: %s", arg);
+ }
+ parse->infile = arg;
+ break;
+
+ default:
+ return ARGP_ERR_UNKNOWN;
+ }
+
+ return 0;
+}
+
+// Parse our "daffodil parse" command line interface
+
+static error_t
+parse_daffodil_parse_cli(struct argp_state *state)
+{
+ int argc = state->argc - state->next + 1;
+ char **argv = &state->argv[state->next - 1];
+ char * old_cmd = argv[0];
+ char new_cmd[strlen(state->name) + strlen(" parse") + 1];
+
+ sprintf(new_cmd, "%s parse", state->name);
+ argv[0] = new_cmd;
+
+ error_t status = argp_parse(&parse_argp, argc, argv, ARGP_IN_ORDER, &argc,
+ &daffodil_parse);
+
+ argv[0] = old_cmd;
+ state->next += argc - 1;
+
+ return status;
+}
+
+// Initialize our "daffodil unparse" CLI options
+
+struct daffodil_unparse_cli daffodil_unparse = {
+ "xml", // default infoset type
+ "-", // default infile
+ "-", // default outfile
+};
+
+static const struct argp_option unparse_options[] = {
+ {"infoset-type", 'I', "<infoset_type>", 0,
+ "Infoset type to unparse. Must be 'xml'"},
+
+ {"output", 'o', "<file>", 0,
+ "Write output to file. If not given or is -, output is written to "
+ "standard output"},
+
+ {0}};
+
+static error_t unparse_handler(int key, char *arg, struct argp_state *state);
+
+static const char unparse_args_doc[] = "[infile]";
+
+static const char unparse_doc[] =
+ "\n"
+ "Unparse an infoset file using a DFDL schema\n"
+ "\n"
+ "Unparse Options:"
+ "\v"
+ " Trailing arguments:\n"
+ " infile (not required) input file to unparse. If not specified, or "
+ "a value of -, reads from stdin";
+
+static const struct argp unparse_argp = {
+ unparse_options, // array of CLI options
+ unparse_handler, // function to get these CLI options
+ unparse_args_doc, // short usage documentation
+ unparse_doc, // long help documentation
+};
+
+// Handle callbacks to get our "daffodil unparse" CLI options
+
+static error_t
+unparse_handler(int key, char *arg, struct argp_state *state)
+{
+ struct daffodil_unparse_cli *unparse = state->input;
+
+ switch (key)
+ {
+ case 'I':
+ unparse->infoset_type = arg;
+ break;
+
+ case 'o':
+ unparse->outfile = arg;
+ break;
+
+ case ARGP_KEY_ARG:
+ if (state->arg_num)
+ {
+ argp_error(state, "too many arguments: %s", arg);
+ }
+ unparse->infile = arg;
+ break;
+
+ default:
+ return ARGP_ERR_UNKNOWN;
+ }
+
+ return 0;
+}
+
+// Parse our "daffodil unparse" command line interface
+
+static error_t
+parse_daffodil_unparse_cli(struct argp_state *state)
+{
+ int argc = state->argc - state->next + 1;
+ char **argv = &state->argv[state->next - 1];
+ char * old_cmd = argv[0];
+ char new_cmd[strlen(state->name) + strlen(" unparse") + 1];
+
+ sprintf(new_cmd, "%s unparse", state->name);
+ argv[0] = new_cmd;
+
+ error_t status = argp_parse(&unparse_argp, argc, argv, ARGP_IN_ORDER,
&argc,
+ &daffodil_unparse);
+
+ argv[0] = old_cmd;
+ state->next += argc - 1;
+
+ return status;
+}
+
+// Initialize our "daffodil" CLI options
+
+struct daffodil_cli daffodil_cli = {
+ DAFFODIL_NONE, // default subcommand
+ 0, // default verbosity
+};
+
+static const struct argp_option daffodil_options[] = {
+ {"verbose", 'v', 0, 0, "Increment verbosity level, one level for each -v",
+ -1},
+
+ {0}};
+
+static error_t daffodil_handler(int key, char *arg, struct argp_state *state);
+
+static const char daffodil_args_doc[] = "<subcommand> [SUBCOMMAND_OPTION...]";
+
+static const char daffodil_doc[] =
+ "\n"
+ "Global Options:"
+ "\v"
+ "Subcommands:\n"
+ " parse Parse data to a DFDL infoset\n"
+ " unparse Unparse a DFDL infoset\n"
+ "\n"
+ "Run 'daffodil <subcommand> --help' for subcommand specific options";
+
+static const struct argp daffodil_argp = {
+ daffodil_options, // array of CLI options
+ daffodil_handler, // function to get these CLI options
+ daffodil_args_doc, // short usage documentation
+ daffodil_doc, // long help documentation
+};
+
+// Handle callbacks to get our "daffodil" CLI options
+
+static error_t
+daffodil_handler(int key, char *arg, struct argp_state *state)
+{
+ struct daffodil_cli *daffodil = state->input;
+ error_t status = 0;
+
+ switch (key)
+ {
+ case 'v':
+ daffodil->verbosity++;
+ break;
+
+ case ARGP_KEY_ARG:
+ if (strcmp(arg, "parse") == 0)
+ {
+ daffodil->subcommand = DAFFODIL_PARSE;
+ status = parse_daffodil_parse_cli(state);
Review comment:
We are indeed using the argp library to shorten the code here (libarp's
a part of the GNU C library). Argp is much better than the getopt library for
complicated CLI requirements. I tried many variations of these names to keep
them as clear as possible but some of these are global names so that's why I
used "daffodil". C really does need a lot of lines of code to implement CLI
parsing even with the argp library but many of these lines are just to
initialize structures.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/runtime2/generators/ParserGenerator.scala
##########
@@ -0,0 +1,314 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2.generators
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.dpath.NodeInfo
+import org.apache.daffodil.dpath.NodeInfo.PrimType
+import org.apache.daffodil.dsom.ElementBase
+import org.apache.daffodil.exceptions.ThrowsSDE
+
+import scala.collection.mutable
+
+/**
+ * Gives an object the ability to generate code.
+ */
+trait ParserGenerator {
+ def generateCode(state: CodeGeneratorState): Unit
+}
+
+/**
+ * Builds up the state of generated code.
+ */
+class CodeGeneratorState extends DFDL.CodeGeneratorState {
+ private val structs = mutable.Stack[ComplexCGState]()
+ private val prototypes = mutable.ArrayBuffer[String]()
+ private val erds = mutable.ArrayBuffer[String]()
+ private val finalStructs = mutable.ArrayBuffer[String]()
+ private val finalImplementation = mutable.ArrayBuffer[String]()
+
+ def addImplementation(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ val initStatements = structs.top.initStatements.mkString("\n")
+ val parserStatements = structs.top.parserStatements.mkString("\n")
+ val unparserStatements = structs.top.unparserStatements.mkString("\n")
+ val prototypeFunctions =
+ s"""static void ${C}_initSelf($C *instance);
+ |static const char *${C}_parseSelf($C *instance, const PState
*pstate);
+ |static const char *${C}_unparseSelf(const $C *instance, const UState
*ustate);""".stripMargin
+ prototypes += prototypeFunctions
+ val functions =
+ s"""static void
+ |${C}_initSelf($C *instance)
+ |{
+ |$initStatements
+ |}
+ |
+ |static const char *
+ |${C}_parseSelf($C *instance, const PState *pstate)
+ |{
+ | const char *error_msg = NULL;
+ |$parserStatements
+ | return error_msg;
+ |}
+ |
+ |static const char *
+ |${C}_unparseSelf(const $C *instance, const UState *ustate)
+ |{
+ | const char *error_msg = NULL;
+ |$unparserStatements
+ | return error_msg;
+ |}
+ |""".stripMargin
+ finalImplementation += functions
+ }
+
+ private def defineQNameInit(context: ElementBase): String = {
+ val qname = context.namedQName.toQNameString
+ val xmlns = if (context.namedQName.prefix.isDefined)
s"xmlns:${context.namedQName.prefix.get}" else "xmlns"
+ val ns = context.namedQName.namespace.toStringOrNullIfNoNS
+ // Optimize away xmlns=ns declaration if possible, although this approach
may not be entirely correct
+ val parentOpt = context.enclosingElements.headOption
+ val parentNs = if (parentOpt.isDefined)
parentOpt.get.namedQName.namespace.toStringOrNullIfNoNS
+ val qnameInit = if (ns == null || ns == parentNs)
+ s""" {"$qname"}, // namedQName.name"""
+ else
+ s""" {
+ | "$qname", // namedQName.name
+ | "$xmlns", // namedQName.xmlns
+ | "$ns", // namedQName.ns
+ | },""".stripMargin
+ qnameInit
+ }
+
+ def addComplexTypeERD(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ val count = structs.top.declarations.length
+ val offsetComputations = structs.top.offsetComputations.mkString(",\n")
+ val erdComputations = structs.top.erdComputations.mkString(",\n")
+ val qnameInit = defineQNameInit(context)
+ val complexERD =
+ s"""static const $C ${C}_compute_ERD_offsets;
+ |
+ |static const ptrdiff_t ${C}_offsets[$count] = {
+ |$offsetComputations
+ |};
+ |
+ |static const ERD *${C}_childrenERDs[$count] = {
+ |$erdComputations
+ |};
+ |
+ |static const ERD ${C}_ERD = {
+ |$qnameInit
+ | COMPLEX, // typeCode
+ | $count, // numChildren
+ | ${C}_offsets, // offsets
+ | ${C}_childrenERDs, // childrenERDs
+ | (ERDInitSelf)&${C}_initSelf, // initSelf
+ | (ERDParseSelf)&${C}_parseSelf, // parseSelf
+ | (ERDUnparseSelf)&${C}_unparseSelf, // unparseSelf
+ |};
+ |""".stripMargin
+ erds += complexERD
+ }
+
+ def addStruct(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ val declarations = structs.top.declarations.mkString("\n")
+ val struct =
+ s"""typedef struct $C
+ |{
+ | InfosetBase _base;
+ |$declarations
+ |} $C;
+ |""".stripMargin
+ finalStructs += struct
+ val initStatement = s" instance->_base.erd = &${C}_ERD;"
+ structs.top.initStatements += initStatement
+ }
+
+ def addSimpleTypeStatements(initStatement: String, parseStatement: String,
unparseStatement: String): Unit = {
+ structs.top.initStatements += initStatement
+ structs.top.parserStatements += parseStatement
+ structs.top.unparserStatements += unparseStatement
+ }
+
+ def addComplexTypeStatements(child: ElementBase): Unit = {
+ val C = child.namedQName.local
+ val e = child.name
+ val initStatement = s" ${C}_initSelf(&instance->$e);"
+ val parseStatement =
+ s""" if (error_msg == NULL)
+ | {
+ | error_msg = ${C}_parseSelf(&instance->$e, pstate);
+ | }""".stripMargin
+ val unparseStatement =
+ s""" if (error_msg == NULL)
+ | {
+ | error_msg = ${C}_unparseSelf(&instance->$e, ustate);
+ | }""".stripMargin
+ structs.top.initStatements += initStatement
+ structs.top.parserStatements += parseStatement
+ structs.top.unparserStatements += unparseStatement
+ }
+
+ def pushComplexElement(context: ElementBase): Unit = {
+ val C = context.namedQName.local
+ structs.push(new ComplexCGState(C))
+ }
+
+ def popComplexElement(context: ElementBase): Unit = {
+ structs.pop()
+ }
+
+ def addSimpleTypeERD(context: ElementBase): Unit = {
+ val e = context.namedQName.local
+ val qnameInit = defineQNameInit(context)
+ val typeCode = context.optPrimType.get match {
+ case PrimType.Int => "PRIMITIVE_INT32"
+ case PrimType.String => "PRIMITIVE_STRING"
+ case p: PrimType => context.SDE("PrimType %s not supported yet.",
p.toString)
+ }
+ val erd =
+ s"""static const ERD ${e}_ERD = {
+ |$qnameInit
+ | $typeCode, // typeCode
+ | 0, // numChildren
+ | NULL, // offsets
+ | NULL, // childrenERDs
+ | NULL, // initSelf
+ | NULL, // parseSelf
+ | NULL, // unparseSelf
+ |};
+ |""".stripMargin
+ erds += erd
+ addComputations(context)
+ }
+
+ def addComputations(child: ElementBase): Unit = {
+ val C = structs.top.C
+ val e = child.namedQName.local
+ val offsetComputation = s" (char *)&${C}_compute_ERD_offsets.$e - (char
*)&${C}_compute_ERD_offsets"
+ val erdComputation = s" &${e}_ERD"
+ structs.top.offsetComputations += offsetComputation
+ structs.top.erdComputations += erdComputation
+ }
+
+ def addFieldDeclaration(context: ThrowsSDE, child: ElementBase): Unit = {
+ val definition = if (child.isSimpleType) {
+ import NodeInfo.PrimType
+ child.optPrimType.get match {
+ case PrimType.Long => "int64_t "
+ case PrimType.Int => "int32_t "
+ case x => context.SDE("Unsupported primitive type: " + x)
+ }
+ } else {
+ child.namedQName.local + " "
+ }
+ structs.top.declarations += s" $definition ${child.name};"
+ }
+
+ def viewCodeHeader: String = {
+ val structs = finalStructs.mkString("\n")
+ val header =
+ s"""#ifndef GENERATED_CODE_H
+ |#define GENERATED_CODE_H
+ |
+ |#include "common_runtime.h" // for InfosetBase
+ |#include <stdint.h> // for int32_t
+ |
+ |// Return the root of an infoset to be used for parsing or unparsing
+ |
+ |extern InfosetBase *rootInfoset();
+ |
+ |// Define some infoset structures
+ |
+ |$structs
+ |#endif // GENERATED_CODE_H
+ |""".stripMargin
+ header
+ }
+
+ def viewCodeFile(rootElementName: String): String = {
+ val prototypes = this.prototypes.mkString("\n")
+ val erds = this.erds.mkString("\n")
+ val finalImplementation = this.finalImplementation.mkString("\n")
+ val code =
+ s"""#include "generated_code.h"
+ |#include <endian.h> // for be32toh, htobe32
+ |#include <errno.h> // for errno
+ |#include <stddef.h> // for ptrdiff_t
+ |#include <stdio.h> // for NULL, fread, fwrite, size_t, feof,
ferror, FILE
+ |#include <string.h> // for strerror
+ |
+ |// Prototypes needed for compilation
+ |
+ |$prototypes
+ |
+ |// Metadata singletons
+ |
+ |$erds
+ |// Return the root of an infoset to be used for parsing or unparsing
+ |
+ |InfosetBase *
+ |rootInfoset()
+ |{
+ | static $rootElementName instance;
+ | InfosetBase *root = &instance._base;
+ | ${rootElementName}_ERD.initSelf(root);
+ | return root;
+ |}
+ |
+ |// Methods to initialize, parse, and unparse infoset nodes
+ |
+ |static const char *
+ |eof_or_error_msg(FILE *stream)
+ |{
+ | if (feof(stream))
+ | {
+ | static const char *error_msg = "Got EOF while expecting more
input";
+ | return error_msg;
+ | }
+ | else if (ferror(stream))
Review comment:
I know that strtol sets errno, but I already checked for errno there so
I'm not sure if something else than fread/fwrite might set errno before calling
this function. I'll take a closer look.
##########
File path: build.sbt
##########
@@ -43,6 +46,32 @@ lazy val runtime1 = Project("daffodil-runtime1",
file("daffodil-runtime1
.dependsOn(io, lib % "test->test", udf, macroLib
% "compile-internal, test-internal")
.settings(commonSettings, usesMacros)
+val runtime2StaticLib = Library("libruntime2.a")
+lazy val runtime2 = Project("daffodil-runtime2",
file("daffodil-runtime2")).configs(IntegrationTest)
Review comment:
So if I understand correctly, you would like to rename some projects
like this:
- rename `daffodil-core` to `daffodil-schema-compiler`
- leave `daffodil-lib` alone
- rename `daffodil-runtime1` to `daffodil-backend-scala-parser`
- rename `daffodil-runtime1-unparser` to `daffodil-backend-scala-unparser`
- rename `daffodil-runtime2` to `daffodil-backend-generator-c`
I think those names are indeed more obvious to newcomers, although I would
move `-scala` to the end to be consistent with `daffodil-backend-generator-c`
(I like symmetric names). Who knows, if the static runtime C code (not the
dynamically generated runtime C code) grows large enough, we might refactor
these C source files from `daffodil-backend-generator-c` to
`daffodil-backend-parser-c` and `daffodil-backend-unparser-c` too.
I'll wait for more developers to weight in on the proposed renaming before I
rename anything, though. We can do the renaming later too since we'll merge
the pull request into its own separate branch to let multiple developers work
on it before we merge it into the main branch.
##########
File path: daffodil-runtime2/src/main/c/common_runtime.c
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
Review comment:
Agreed, I remembered what you said here and raised the same point for
wider discussion in an earlier comment.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/runtime2/Runtime2DataProcessor.scala
##########
@@ -0,0 +1,211 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import java.io.File
+import java.io.InputStream
+import java.io.OutputStream
+
+import org.apache.daffodil.api.DFDL
+import org.apache.daffodil.api.DaffodilTunables
+import org.apache.daffodil.api.DataLocation
+import org.apache.daffodil.api.ValidationMode
+import org.apache.daffodil.externalvars.Binding
+import org.apache.daffodil.processors.Failure
+import org.apache.daffodil.processors.ProcessorResult
+import org.apache.daffodil.processors.Success
+import org.apache.daffodil.processors.VariableMap
+import org.apache.daffodil.processors.WithDiagnosticsImpl
+import org.apache.daffodil.processors.parsers.ParseError
+import org.apache.daffodil.processors.unparsers.UnparseError
+import org.apache.daffodil.util.Maybe
+import org.apache.daffodil.util.Maybe.Nope
+import os.Pipe
+
+/**
+ * Effectively a scala proxy object that does its work via the underlying
C-code.
+ * Will need to consider how to use features of underlying C-code to get
infoset,
+ * walk infoset, generate XML for use by TDML tests.
+ */
+class Runtime2DataProcessor(executableFile: os.Path) extends
DFDL.DataProcessorBase {
+ /**
+ * Returns a data processor with all the same state, but the validation mode
changed to that of the argument.
+ *
+ * Note that the default validation mode is "off", that is, no validation is
performed.
+ */
+ override def withValidationMode(mode: ValidationMode.Type):
DFDL.DataProcessor = ???
+
+ override def withTunable(name: String, value: String): DFDL.DataProcessor =
???
+
+ override def withTunables(tunables: Map[String, String]): DFDL.DataProcessor
= ???
+
+ override def withExternalVariables(extVars: Map[String, String]):
DFDL.DataProcessor = ???
+
+ override def withExternalVariables(extVars: File): DFDL.DataProcessor = ???
+
+ override def withExternalVariables(extVars: Seq[Binding]):
DFDL.DataProcessor = ???
+
+ override def validationMode: ValidationMode.Type = ???
+
+ override def getTunables(): DaffodilTunables = ???
+
+ override def save(output: DFDL.Output): Unit = ???
+
+ override def variableMap: VariableMap = ???
+
+ override def setValidationMode(mode: ValidationMode.Type): Unit = ???
+
+ override def setExternalVariables(extVars: Map[String, String]): Unit = ???
+
+ override def setExternalVariables(extVars: File): Unit = ???
+
+ override def setExternalVariables(extVars: File, tunable: DaffodilTunables):
Unit = ???
+
+ override def setExternalVariables(extVars: Seq[Binding]): Unit = ???
+
+ override def setTunable(tunable: String, value: String): Unit = ???
+
+ override def setTunables(tunables: Map[String, String]): Unit = ???
+
+ /**
+ * Returns an object which contains the result, and/or diagnostic
information.
+ */
+ def parse(input: InputStream): ParseResult = {
Review comment:
I agree that we can move the snap compilation logic to
Runtime2TDMLDataProcessor so we may not need a Runtime2DataProcessor.
I haven't seen anyone discuss my own proposal: add an entirely new daffodil
generate subcommand. Here're all the CLI commands everyone's already proposed
along with my own proposal:
daffodil runtime2 -s schema.dfdl.xsd -o schema.bin
daffodil save-parser -s schema.dfdl.xsd outfile
daffodil generate -s schema.dfdl.xsd outdir/
I think that generate is sufficiently different from save-parser in
semantics that it should be its own command. Right now save-parser takes a
single output file and writes to it, but generate would take a directory and
write multiple files into it to give people the choice of an executable with a
CLI and main method already compiled in or headers and libraries that people
can add to their own application. I also think generate is a more obvious name
than runtime2 and allows the possibility of different backends using the same
"generate" command to generate C, Java, or VHDL files (we can use a "runtime"
tunable or a CLI option when we have multiple generator backends).
I'm also intrigued by the suggestion to allow the Daffodil Scala CLI to ask
the different backends for their CLI options. The generate command can offer
different CLI options depending on the backend in that case.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/runtime2/GeneratedCodeCompiler.scala
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.daffodil.runtime2
+
+import org.apache.commons.io.FileUtils
+import org.apache.daffodil.compiler.ProcessorFactory
+import org.apache.daffodil.dsom.SchemaDefinitionError
+import org.apache.daffodil.runtime2.generators.CodeGeneratorState
+import org.apache.daffodil.util.Misc
+import os.Pipe
+
+class GeneratedCodeCompiler(pf: ProcessorFactory) {
Review comment:
What do people think? Should we compile the static runtime C code when
we build Daffodil and distribute the static lib in the zip/tar/rpm or should we
put the C source files in src/main/resources/c/libruntime2 in order to extract
them from the jar when it's time to compile all of the C source files for the
first time on a user's machine? I want to keep things as simple as possible
until requirements become clearer, but we definitely should come to a consensus
on when to compile the static library.
##########
File path:
daffodil-core/src/main/scala/org/apache/daffodil/grammar/GrammarTerm.scala
##########
@@ -51,7 +52,8 @@ import org.apache.daffodil.dsom.Term
abstract class Gram(contextArg: SchemaComponent)
extends OOLAGHostImpl(contextArg)
with BasicComponent
- with GramRuntime1Mixin {
+ with GramRuntime1Mixin
+ with GramRuntime2Mixin {
Review comment:
I like Brandon's suggestion to turn generateCode() from a method of Gram
(through a mixin trait) into a simple function in daffodil-backend-generator-c
that takes Gram as an argument. However, Gram's generateCode() is not the only
method we have added to traits/classes in runtime1. I'll have to evaluate how
easy it would be to move everything to daffodil-backend-generator-c. I'll
start with Gram's generatorCode() at least so we can see what else is left to
move.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
