stevedlawrence closed pull request #3: Add user related pages that were
previously stored on the wiki
URL: https://github.com/apache/incubator-daffodil-site/pull/3
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/site/_includes/themes/apache/_navigation.html
b/site/_includes/themes/apache/_navigation.html
index 5c4f600..2c53cd6 100644
--- a/site/_includes/themes/apache/_navigation.html
+++ b/site/_includes/themes/apache/_navigation.html
@@ -18,9 +18,13 @@
<li id="documentation">
<a href="#" data-toggle="dropdown"
class="dropdown-toggle">Docs<b class="caret"></b></a>
<ul class="dropdown-menu dropdown-left">
+ <li><a href="/getting-started/">Getting Started</a></li>
+ <li><a href="/examples/">Examples</a></li>
<li><a href="/docs/latest/javadoc/">Java API</a></li>
<li><a href="/docs/latest/scaladoc/">Scala API</a></li>
<li><a href="/docs/dfdl/">DFDL Specification</a></li>
+ <li><a href="/unsupported/">Unsupported Features</a></li>
+ <li><a href="/faq/">Frequently Asked Questions</a></li>
</ul>
</li>
<li id="community">
@@ -33,19 +37,19 @@
<li id="development">
<a href="#" data-toggle="dropdown"
class="dropdown-toggle">Development<b class="caret"></b></a>
<ul class="dropdown-menu dropdown-left">
- <li><a
href="https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow";><i
class="external"></i>Contributor Workflow</a></li>
- <li><a href="https://github.com/apache/incubator-daffodil";><i
class="external"></i>GitHub</a></li>
- <li><a
href="https://issues.apache.org/jira/projects/DAFFODIL/";><i
class="external"></i>JIRA</a></li>
+ <li><a class="external"
href="https://cwiki.apache.org/confluence/display/DAFFODIL/Code+Contributor+Workflow";>Contributor
Workflow</a></li>
+ <li><a class="external"
href="https://github.com/apache/incubator-daffodil";>GitHub</a></li>
+ <li><a class="external"
href="https://issues.apache.org/jira/projects/DAFFODIL/";>JIRA</a></li>
</ul>
</li>
<li id="apache">
<a href="#" data-toggle="dropdown"
class="dropdown-toggle">Apache<b class="caret"></b></a>
<ul class="dropdown-menu">
- <li><a href="http://www.apache.org/";><i
class="external"></i>Apache Software Foundation</a></li>
- <li><a href="http://www.apache.org/licenses/";><i
class="external"></i>License</a></li>
- <li><a href="http://www.apache.org/security";><i
class="external"></i>Security</a></li>
- <li><a
href="http://www.apache.org/foundation/sponsorship.html";><i
class="external"></i>Sponsorship</a></li>
- <li><a href="http://www.apache.org/foundation/thanks.html";><i
class="external"></i>Thanks</a></li>
+ <li><a class="external" href="http://www.apache.org/";>Apache
Software Foundation</a></li>
+ <li><a class="external"
href="http://www.apache.org/licenses/";>License</a></li>
+ <li><a class="external"
href="http://www.apache.org/security";>Security</a></li>
+ <li><a class="external"
href="http://www.apache.org/foundation/sponsorship.html";>Sponsorship</a></li>
+ <li><a class="external"
href="http://www.apache.org/foundation/thanks.html";>Thanks</a></li>
</ul>
</li>
</ul>
diff --git a/site/_layouts/release.html b/site/_layouts/release.html
index 4ac871d..6b4dad4 100644
--- a/site/_layouts/release.html
+++ b/site/_layouts/release.html
@@ -82,6 +82,6 @@ <h3>API Dependency</h3>
</div>
</div>
-<h3>Release Notes</h3>
+<h3 id="release-notes">Release Notes</h3>
{{ content }}
diff --git a/site/_plugins/Features.rb b/site/_plugins/Features.rb
new file mode 100644
index 0000000..10e1e16
--- /dev/null
+++ b/site/_plugins/Features.rb
@@ -0,0 +1,35 @@
+module Jekyll
+ class OkTag < Liquid::Tag
+ def initialize(tag_name, text, tokens)
+ super
+ end
+
+ def render(context)
+ "<i class='glyphicon glyphicon-ok' style='color: #00d000;'></i>"
+ end
+ end
+
+ class ErrorTag < Liquid::Tag
+ def initialize(tag_name, text, tokens)
+ super
+ end
+
+ def render(context)
+ "<i class='glyphicon glyphicon-remove' style='color: #d00000;'></i>"
+ end
+ end
+
+ class WarningTag < Liquid::Tag
+ def initialize(tag_name, text, tokens)
+ super
+ end
+
+ def render(context)
+ "<i class='glyphicon glyphicon-minus' style='color: #efcb00;'></i>"
+ end
+ end
+end
+
+Liquid::Template.register_tag('ok', Jekyll::OkTag)
+Liquid::Template.register_tag('err', Jekyll::ErrorTag)
+Liquid::Template.register_tag('warn', Jekyll::WarningTag)
diff --git a/site/assets/themes/apache/css/style.css
b/site/assets/themes/apache/css/style.css
index 96690e0..74849fd 100644
--- a/site/assets/themes/apache/css/style.css
+++ b/site/assets/themes/apache/css/style.css
@@ -8,6 +8,16 @@ body {
font-size: 16px;
}
+/* this makes it so anchors are slightly below the navbar so they
+ * aren't covered up when linked */
+*[id]:before {
+ display:block;
+ content: " ";
+ margin-top: -75px;
+ height: 75px;
+ visibility: hidden;
+}
+
h1,
h2,
h3,
@@ -393,7 +403,7 @@ hr {
}
}
-i.external {
+a.external:before {
position: relative;
top: 1px;
display: inline-block;
@@ -406,7 +416,7 @@ i.external {
margin-right: 5px;
}
-i.external:before {
+a.external:before {
content: "\e066";
}
@@ -416,3 +426,13 @@ i.external:before {
border-top-right-radius: 0px;
border-color: #dddddd;
}
+
+dd {
+ margin-top: 10px;
+ margin-left: 30px;
+}
+
+dt code {
+ color: inherit;
+ background-color: inherit;
+}
diff --git a/site/cli.md b/site/cli.md
new file mode 100644
index 0000000..1322847
--- /dev/null
+++ b/site/cli.md
@@ -0,0 +1,370 @@
+---
+layout: page
+title: Command Line Interface
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Command Line Interface
+
+The binary Daffodil [release](/release) contain a ``/bin`` directory that
contains two scripts: ``daffodil.bat`` for Windows and ``daffodil`` for Linux.
These files must be executed on the command line. The general usage is:
+
+ daffodil [GLOBAL_OPTIONS] <subcommand> [SUBCOMMAND_OPTIONS]
+
+The available subcommands are: [parse](#parse-subcommand),
[unparse](#unparse-subcommand), [save-parser](#save-parser-subcommand),
[test](#test-subcommand), and [performance](#performance-subcommand).
+
+### Environment Variables
+
+Setting environment variables may be necessary to allow for importing/includes
to work and when running TDML files.
+
+``DAFFODIL_CLASSPATH``
+
+ : The Daffodil CLI will look on the classpath for includes and imports,
jars containing schemas, and some TDML files. To define additional directories
specific to Daffodil to look for files, set the ``DAFFODIL_CLASSPATH``
environment variable, for example:
+
+ $ export DAFFODIL_CLASSPATH="/path/to/imports/:/path/to/includes/"
+
+ In addition to defining directories to search for imports and includes,
you can add a CatalogManager.properties file to the ``DAFFODIL_CLASSPATH`` to
direct Daffodil to a relative path location of a user XML Catalog.
+
+``DAFFODIL_JAVA_OPTS``
+
+ : If you need to specify java options specific to Daffodil, you can set the
``DAFFODIL_JAVA_OPTS`` environment variable. If not specified, the
``JAVA_OPTS`` environment variable will be used. If that is not specified,
reasonable defaults for Daffodil will be used.
+
+### Global Options
+
+``-d, --debug [FILE]``
+
+ : Enable the interactive debugger. See the [Interactive
Debugger](/debugger) documentation for more information.
+
+ The optional ``FILE`` argument contains a list of debugger commands that
are provided to the debugger as they were typed by the user.
+
+ This option cannot be used with the ``--trace`` option.
+
+``-t, --trace``
+
+ : Enable a trace mode. This mode prints out helpful information during
every stage of parsing.
+
+ This option cannot be used with the ``--debug`` option.
+
+``-v, --verbose``
+
+ : Enable verbose mode and increment verbosity level. Each additional v
provides a new level of information.
+
+``--version``
+
+ : Display the version of Daffodil.
+
+``--help``
+
+ : Display help message.
+
+### Parse Subcommand
+
+Parse a file, using either a DFDL schema or a saved parser.
+
+#### Usage
+
+ daffodil parse (-s <schema> [-r [{<namespace>}]<root>] [-p <path>] | -P
<parser>)
+ [-D[{namespace}]<variable>=<value>...]
[-T<tunable>=<value>] [-o <output>]
+ [-V [mode]] [-c <file>] [-I <infoset_type>] [infile]
+
+#### Options
+
+``-c, --config FILE``
+
+ : XML file containing configuration items, such as external variables or
Daffodil tunables. See Configuration File for details on the file format.
+
+``-D [{NAMESPACE}]VARIABLE=VALUE``
+
+ : Variables to be used when parsing. ``VARIABLE`` can be prefixed with
``{NAMESPACE}`` to define which namespace the variable belongs in, for example:
+
+ ```-D{http://example.com}var1=var```
+
+``-I, --infoset-type TYPE``
+
+ : Infoset type to output. ``TYPE`` must be one of ``xml``, ``scala-xml``,
``json``, ``jdom``, or ``null``. Defaults to ``xml`` if not provided.
+
+``-o, --output FILE``
+
+ : Write output to a given ``FILE``. If the option is not given or ``FILE``
is -, output is written to standard out.
+
+``-P, --parser FILE``
+
+ : Use a previously saved parser, created using the save-parser subcommand.
This option cannot be used with the ``--schema`` option or with the
``--validate`` option set to ``on``.
+
+``-p, --path PATH``
+
+ : The path to the node from the root element to create the parser from.
+
+``-r, --root [{NAMESPACE}]ROOT``
+
+ : The root element of the DFDL schema to use. This must be one of the
top-level elements of the schema defined with ``--schema``. This requires the
``--schema`` option to be defined. If not supplied, the first element of the
schema defined with ``--schema`` is used. A namespace may be specified by
prefixing ``ROOT`` with ``{NAMESPACE}``.
+
+``-s, --schema FILE``
+
+ : The annotated DFDL schema to use to create the parser. This option cannot
be used with the ``--parser`` option.
+
+``-T TUNABLE=VALUE``
+
+ : Modify Daffodil configuration options to change parsing behavior. See
Tunable Parameters for the list of tunable parameters.
+
+``-V, --validate [MODE]``
+
+ : The validation mode. ``MODE`` must be one of ``on``, ``limited`` or
``off``. If ``MODE`` is not provided, defaults to ``on``. If ``--validate`` is
not provided, defaults to ``off``. ``MODE`` cannot be ``on`` when used with the
``--parser`` option.
+
+``[INFILE]``
+
+ : Input file to parse. If not specified, or is a value of -, reads from
standard in. If supplied, this must be the last option on the command line.
+
+``--help``
+
+ : Display help message.
+
+#### Example
+
+ $ daffodil parse -s csv.dfdl.xsd test_file.csv
+
+### Unparse Subcommand
+
+Unparse an infoset file, using either a DFDL schema or a saved parser.
+
+#### Usage
+
+ daffodil unparse (-s <schema> [-r [{<namespace>}]<root>] [-p <path>] | -P
<parser>)
+ [-D[{<namespace>}]<variable>=<value>...]
[-T<tunable>=<value>] [-o <output>]
+ [-V [mode]] [-c <file>] [-I <infoset_type>] [infile]
+
+#### Options
+
+``-c, --config FILE``
+
+ : XML file containing configuration items, such as external variables or
Daffodil tunables. See Configuration File for details on the file format.
+
+``-D [{NAMESPACE}]VARIABLE=VALUE``
+
+ : Variables to be used when unparsing. ``VARIABLE`` can be prefixed with
``{NAMESPACE}`` to define which namespace the variable belongs in, for example:
+
+ ```-D{http://example.com}var1=var```
+
+``-I, --infoset-type TYPE``
+
+ : Infoset type to unparse. ``TYPE`` must be one of ``xml``, ``scala-xml``,
``json``, or ``jdom``. Defaults to ``xml`` if not provided.
+
+``-o, --output FILE``
+
+ : Write output to a give ``FILE``. If the option is not given or ``FILE``
is -, output is written to standard out.
+
+``-P, --parser FILE``
+
+ : Use a previously saved parser, created using the save-parser subcommand.
This option cannot be used with the ``--schema`` option or with the
``--validate`` option set to ``on``.
+
+``-p, --path PATH``
+
+ : The path to the node from the root element to create the parser from.
+
+``-r, --root [{NAMESPACE}]ROOT``
+
+ : The root element of the DFDL schema to use. This must be one of the
top-level elements of the schema defined with ``--schema``. This requires the
``--schema`` option to be defined. If not supplied, the first element of the
schema defined with ``--schema`` is used. A namespace may be specified by
prefixing the ``ROOT`` with {NAMEAPSCE}.
+
+``-s, --schema FILE``
+
+ : The annotated DFDL schema to use to create the parser. This option cannot
be used with the ``--parser`` option.
+
+``-T TUNABLE=VALUE``
+
+ : Modify Daffodil configuration options to change parsing behavior. See
Tunable Parameters for the list of tunable parameters.
+
+``-V, --validate [MODE]``
+
+ : The validation mode. ``MODE`` must be one of ``on``, ``limited`` or
``off``. If ``MODE`` is not provided, defaults to ``on``. If ``--validate`` is
not provided, defaults to ``off``. ``MODE`` cannot be ``on`` when used with the
``--parser`` option.
+
+``[INFILE]``
+
+ : Input file to unparse. If not specified, or is a value of -, reads from
standard in. If supplied, this must be the last option on the command line.
+
+``--help``
+ : Display help message.
+
+#### Example
+
+ $ daffodil unparse -s csv.dfdl.xsd test_file.infoset
+
+### Save Parser Subcommand
+
+Save a parser that can be reused for parsing and unparsing.
+
+#### Usage
+
+ daffodil save-parser -s <schema> [-r [{namespace}]<root>] [-p <path>]
[outfile]
+
+#### Options
+
+``-c, --config FILE``
+
+ : XML file containing configuration items, such as external variables or
Daffodil tunables. See Configuration File for details on the file format.
+
+``-D [{NAMESPACE}]VARIABLE=VALUE``
+
+ : Variables to be used when parsing. ``VARIABLE`` can be prefixed with
``{NAMESPACE}`` to define which namespace the variable belongs in, for example:
+
+ ```-D{http://example.com}var1=var```
+
+``-p, --path PATH``
+
+ : The path to the node from the root element to create the parser from.
+
+``-r, --root [{NAMESPACE}]ROOT``
+
+ : The root element of the DFDL schema to use. This must be one of the
top-level elements of the schema defined with ``--schema``. This requires the
``--schema`` option to be defined. If not supplied, the first element of the
schema defined with ``--schema`` is used. A namespace may be specified by
prefixing ``ROOT`` with ``{NAMESPACE}``.
+
+``-s, --schema FILE``
+
+ : The annotated DFDL schema to use to create the parser. This option must
be supplied.
+
+``-T TUNABLE=VALUE``
+
+ : Modify Daffodil configuration options to change parsing behavior. See
Tunable Parameters for the list of tunable parameters.
+
+``-V, --validate MODE``
+
+ : The validation mode. ``MODE`` must be either ``limited`` or ``off``.
``MODE`` is required. If ``--validate`` is not present, defaults to ``off``.
``MODE`` cannot be set to ``on`` as possible in other commands.
+
+``[OUTFILE]``
+
+ : Write the parser to a give file. If the option is not given or is -,
output is written to standard out. If supplied, this must be the last option on
the command line.
+
+``--help``
+
+ : Display help message.
+
+#### Example
+
+ $ daffodil save-parser -s csv.dfdl.xsd csv_parser.xml
+
+### Test Subcommand
+
+List or execute tests in a TDML file.
+
+#### Usage
+
+ daffodil test [-l] [-r] [-i] <tdmlfile> [testname...]
+
+#### Options
+
+``-i, --info``
+
+ : Increment test result information output level, one level for each
occurrence of -i.
+
+``-l, --list``
+
+ : Show names and descriptions in a TDML file instead of running them.
+
+``-r, --regex``
+
+ : Treat ``TESTNAME``s as regular expressions.
+
+``TDMLFILE``
+
+ : Test Data Markup Language (TDML) file.
+
+``[TESTNAME...]``
+
+ : Name of one or more test case in the TDML file. If not given, all tests
in ``TDMLFILE`` are run.
+
+``--help``
+
+ : Display help message.
+
+#### Example
+
+ $ daffodil test csv.tdml
+
+### Performance Subcommand
+
+Run a performance test (parse or unparse), using either a DFDL schema or a
saved parser
+
+#### Usage
+
+ daffodil performance (-s <schema> [-r [{namespace}]<root>] [-p <path>] |
-P <parser)
+ [-u] [-V[mode]] [-N <number>] [-t <threads>]
+ [-D[{namespace}]<variable>=<value>] [-I
<infoset_type>] <infile>
+
+#### Options
+
+``-c, --config FILE``
+
+ : XML file containing configuration items, such as external variables or
Daffodil tunables. See Configuration File for details on the file format.
+
+``-D [{NAMESPACE}]VARIABLE=VALUE``
+
+ : Variables to be used when parsing. ``VARIABLE`` can be prefixed with
``{NAMESPACE}`` to define which namespace the variable belongs in, for example:
+
+ ```-D{http://example.com}var1=var```
+
+``-I, --infoset-type TYPE``
+
+ : Infoset type to parse/unparse. ``TYPE`` must be one of ``xml``,
``scala-xml``, ``json``, ``jdom``, or ``null``. Defaults to ``xml`` if not
provided. Note that ``null`` is not valid if the ``--unparse`` option is
provided.
+
+``-N, --number NUMBER``
+
+ : Total number of files to process. Defaults to 1.
+
+``-P, --parser FILE``
+
+ : Use a previously saved parser, created using the save-parser subcommand.
This option cannot be used with the ``--schema`` option or with the
``--validate`` option set to ``on``.
+
+``-p, --path PATH``
+
+ : The path to the node from the root element to create the parser from.
+
+``-r, --root [{NAMESPACE}]ROOT``
+
+ : The root element of the DFDL schema to use. This must be one of the
top-level elements of the schema defined with ``--schema``. This requires the
``--schema`` option to be defined. If not supplied, the first element of the
schema defined with ``--schema`` is used. A namespace may be specified by
prefixing ``ROOT`` with ``{NAMESPACE}``.
+
+``-s, --schema FILE``
+
+ : The annotated DFDL schema to use to create the parser. This option cannot
be used with the ``--parser`` option.
+
+``-t, --threads THREADS``
+
+ : The number of threads to use. Defaults to 1.
+
+``-T TUNABLE=VALUE``
+
+ : Modify Daffodil configuration options to change processing behavior. See
Tunable Parameters for the list of tunable parameters.
+
+``-u, --unparse``
+
+ : Perform unparse instead of parse for performance.
+
+``-V, --validate [MODE]``
+
+ : The validation mode. ``on``, ``limited`` or ``off``. If ``MODE`` is not
provided, defaults to ``on``. If ``--validate`` is not provided, defaults to
``off``. ``MODE`` cannot be ``on`` when used with the ``--parser`` option.
+
+``[INFILE]``
+
+ : Input file or directory containing files to process
+
+``--help``
+
+ : Display help message.
+
+#### Example
+
+ $ daffodil performance -s csv.dfdl.xsd -N 1000 -t 5 test_file.csv
diff --git a/site/configuration.md b/site/configuration.md
new file mode 100644
index 0000000..8cb469d
--- /dev/null
+++ b/site/configuration.md
@@ -0,0 +1,123 @@
+---
+layout: page
+title: Configuration
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Configuration
+
+Daffodil has the capability to read in parameters via an external XML file,
defined by the
[dafext.xsd](https://github.com/apache/incubator-daffodil/blob/master/daffodil-propgen/src/main/resources/edu/illinois/ncsa/daffodil/xsd/dafext.xsd)
schema. Below are the parameters that can be defined.
+
+### External Variables
+
+External variables can be defined using the ``externalVariablesBindings`` tag.
For example, the following configuration file defines two variables, ``var1``
and ``var2``, in the ``http://example.com`` namespace.
+
+``` xml
+<?xml version="1.0" encoding="UTF-8"?>
+<dfdlConfig xmlns="http://www.w3.org/2001/XMLSchema";
xmlns:ex="http://example.com";>
+ <externalVariableBindings>
+ <bind name="ex:var1">-9</bind>
+ <bind name="ex:var2">Foo</bind>
+ </externalVariableBindings>
+</dfdlConfig>
+```
+
+The following defined in a DFDL schema would allow the external variables to
be set or overridden:
+
+``` xml
+<dfdl:defineVariable name="var1" external="true"
type="xsd:int">1</dfdl:defineVariable>
+<dfdl:defineVariable name="var2" external="true"
type="xsd:string">Bar</dfdl:defineVariable>
+```
+
+### Tunable Parameters
+
+Tunable parameters can be modified to change Daffodil schema compilation and
data parsing properties. For example, the following sets the
``maxOccursBounds`` tunable to 1024:
+
+``` xml
+<?xml version="1.0" encoding="UTF-8"?>
+<dfdlConfig xmlns="http://www.w3.org/2001/XMLSchema";
xmlns:ex="http://example.com";>
+ <tunables>
+ <maxOccursBounds>1024</maxOccursBounds>
+ </tunables>
+</dfdlConfig>
+```
+
+Below is a description of the available tunable parameters:
+
+``generatedNamespacePrefixStem``
+
+ : If elementFormDefault is qualified, but no prefix is associated with the
targetNamespace, a prefix will be created using this value, with numbers
appended if necessary to ensure uniqueness. Defaults to "tns".
+
+``inputFileMemoryMapLowThreshold``
+
+ : In certain I/O optimized situations (text-only,
encodingErrorPolicy='replace', fixed-width encoding) input files larger than
this will be mmapped. Input files smaller than this will be simply read using
ordinary I/O (because for small files that is just faster). This exists because
mmap is more expensive than ordinary I/O for small files. Defaults to 225.
+
+``maxBinaryDecimalVirtualPoint``
+
+ : An upper limit on the value of the dfdl:binaryDecimalVirtualPoint
property. Defaults to 200.
+
+``maxFieldContentLengthInBytes``
+
+ : A maximum limit for various daffodil I/O properties, including the
maximim size of a simple element and maximum regular expression characters to
match. Defaults to 220.
+
+``maxLengthForVariableLengthDelimiterDisplay``
+
+ : When unexpected text is found where a delimiter is expected, this is the
maximum number of bytes (characters) to display when the expected delimiter is
a variable length delimiter. Defaults to 10.
+
+``maxOccursBounds``
+
+ : A maximum limit for the number of repeats of array elements. Defaults to
1024.
+
+``maxSkipLengthInBytes``
+
+ : A maximum limit for the number of bytes that can be skipped in a skip
region. Defaults to 1024.
+
+``minBinaryDecimalVirtualPoint``
+
+ : A lower limit on the value of the dfdl:binaryDecimalVirtualPoint
property. Defaults to -200.
+
+``parseUnparsePolicy``
+
+ : Whether to compile a schema to support parsing ("parseOnly"), unparsing
("unparseOnly"), both parsing and unparsing ("both"), or to use the
daf:parseUnparsePolicy property from the root node ("schema"). Defaults to
"both".
+
+``requireBitOrderProperty``
+
+ : If true, require that the bitOrder property is specified. If false, use a
default value for bitOrder if not defined in a schema. Defaults to false.
+
+``requireEncodingErrorPolicyProperty``
+
+ : If true, require that the encodingErrorPolicy property is specified. If
false, use a default value if not defined in a schema. Defaults to false.
+
+``unqualifiedPathStepPolicy``
+
+ : Specified how unqualified path steps are resolved. Defaults to
``noNamespace``. Value values are:
+
+ ``noNamespace``
+
+ : Unqualified path steps remain unqualified and will only match
elements in NoNamespace. A prefix must be provided to match namespaced elements.
+
+ ``defaultNamespace``
+
+ : Unqualified path steps will always use the default namespace. If a
default namespace is defined, it is not possible to match a NoNamespace element
with this policy. Because of this, this may not work well with
elementFormDefault="unqualified".
+
+ ``preferDefaultNamespace``
+
+ : Attempt to use the default namespace to resolve an unqualified path
step. If that fails to match an element, then try to resolve using NoNamespace.
diff --git a/site/debugger.md b/site/debugger.md
new file mode 100644
index 0000000..1ea2f2b
--- /dev/null
+++ b/site/debugger.md
@@ -0,0 +1,255 @@
+---
+layout: page
+title: Interactive Debugger
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Interactive Debugger
+
+The Daffodil [Command Line Interface](/cli) (CLI) comes with a built-in
interactive debugger that pauses parsing/unparsing and allows for user
inspection of some internal state. To start Daffodil in debugger mode, provide
the global ``--debug`` option, for example:
+
+ daffodil --debug parse --schema schema.dfdl.xsd input.dat
+
+Enabling the debugger displays a ``(debug)`` prompt, at which point various
commands can be entered to control parsing and inspect Daffodil state. Note
that you can optionally provide a file to the ``--debug`` option. Each line in
the file is provided to the debugger as if it were typed by the user. Below is
the list of commands accepted by the debugger and their descriptions.
+
+``break <element_id>``
+
+ : Create a breakpoint, causing the debugger to stop when the element with
the ``<element_id>`` name is created.
+
+ break foo
+
+ Abbreviation: ``b``
+
+``clear``
+
+ : Clear the screen.
+
+ : Abbreviation: ``c``
+
+``complete``
+
+ : Continue parsing the input data until parsing is complete. All
breakpoints are ignored.
+
+ : Abbreviation: ``comp``
+
+``condition <breakpoint_id> <dfdl_expression>``
+
+ : Set a condition on a specified breakpoint. When a breakpoint is
encountered, the debugger only pauses if the DFDL expression evaluates to
``true``. If the result of the DFDL expressions is not a boolean value, it is
treated as false.
+
+ : Abbreviation: ``cond``
+
+ condition 1 dfdl:occursIndex() eq 3
+
+``continue``
+
+ : Continue parsing the input data until a breakpoint is encountered. At
which point, pause parsing and display a debugger console to the user.
+
+ : Abbreviation: ``c``
+
+``delete <type> <id>``
+
+ : Remove a breakpoint or display. Valid values for ``<type>`` are
``breakpoint`` and ``display``.
+
+ : Abbreviation: ``d``
+
+ delete breakpoint 1
+ delete display 1
+
+``disable <type> <id>``
+
+ : Disable a breakpoint or display. Valid values for ``<type>`` are
``breakpoint`` and ``display``.
+
+ : Abbreviation: ``dis``
+
+ disable breakpoint 1
+ disable display 1
+
+``display <debugger_command>``
+
+ : Execute a debugger command every time a debugger console is displayed to
the user.
+
+ : Abbreviation: ``di``
+
+ display info infoset
+
+``enable <type> <id>``
+
+ : Enable a breakpoint or display. Valid values for ``<type>`` are
``breakpoint`` and ``display``.
+
+ : Abbreviation: ``e``
+
+ enable breakpoint 1
+ enable display 1
+
+``eval <dfdl_expression>``
+
+ : Evaluate a DFDL expression. If the expression evaluates to a complex
element, then the XML representation for the complex element is displayed. If
the expression evaluates to a simple type, then the value of the simple type is
display.
+
+ : Abbreviation: ``ev``
+
+ eval dfdl:occursIndex()
+ eval /ex:file/line[1]
+
+``help [command]``
+
+ : Display help. If a command is given, display help information specific to
that command and its subcommands.
+
+ : Abbreviation: ``h``
+
+``history [outfile]``
+
+ : Display the history of commands. If an argument is given, write the
history to the specified file rather then printing it to the screen.
+
+ : Abbreviation: ``hi``
+
+``info <item>...``
+
+ : Print internal information to the console. ``<item>`` can be specified
multiple times to display multiple pieces of information.
+
+ : Abbreviation: ``i``
+
+ info data infoset
+
+ The valid items are:
+
+ ``arrayIndex``
+
+ : Display the current array limit. Abbreviation: ``ai``
+
+ ``bitLimit``
+
+ : Display the current bit limit. Abbreviation: ``bl``
+
+ ``bitPosition``
+
+ : Display the current bit position. Abbreviation: ``bp``
+
+ ``breakpoints``
+
+ : Display the list of breakpoints. Abbreviation: ``b``
+
+ ``childIndex``
+
+ : Display the child index. Abbreviation: ``ci``
+
+ ``data``
+
+ : Display the input data. Abbreviation: ``d``
+
+ ``delimiterStack``
+
+ : Display the current delimiter stack. Abbreviation: ``ds``
+
+ ``diff``
+
+ : Display the differences from the previous state.
+
+ ``discriminator``
+
+ : Display whether or not a discriminator is set. Abbreviation: ``dis``
+
+ ``displays``
+
+ : Display the current ``display`` expressions. Abbreviation: ``di``
+
+ ``foundDelimiter``
+
+ : Display the most recently found delimiter. Abbreviation: ``fd``
+
+ ``groupIndex``
+
+ : Display the current group index. Abbreviation: ``gi``
+
+ ``infoset``
+
+ : Display the current infoset. Abbreviation: ``i``
+
+ ``occursBounds``
+
+ : Display the current occurs bounds. Abbreviation: ``ob``
+
+ ``parser``
+
+ : Display the current parser. Abbreviation: ``p``
+
+ ``path``
+
+ : Display the current schema component designator/path.
+
+ ``unparser``
+
+ : Display the current unparser. Abbreviation: ``u``
+
+``quit``
+
+ : Immediately abort all processing.
+
+ : Abbreviation: ``q``
+
+``set <setting> <value>``
+ : Change a debugger setting.
+
+ set dataLength -1
+ set wrapLength 50
+ set removeHidden true
+
+ Valid settings are:
+
+ ``breakOnFailure <boolean>``
+
+ : Set whether or not the debugger should break on failures. If set to
``false`` the normal processing occurs. If set to ``true``, any errors cause a
break. Note that due to the backtracking behavior, not all failures are fatal.
Defaults to ``false``.
+
+ ``breakOnlyOnCreation <boolean>``
+
+ : Set whether or not breakpoints should only be evaluated on element
creation. Must be either ``true``/``false`` or ``1``/``0``. If ``true``,
breakpoints only stop on element creation. If ``false``, breakpoints stop
anytime a parser interacts with an element. Defaults to ``true``.
+
+ ``dataLength <integer>``
+
+ : Set the number of bytes to display when displaying input data. If
negative, display all input data. This only affects the ``info data`` command.
Defaults to ``70``.
+
+ ``infosetLines <integer>``
+
+ : Set the maximum number of lines to display when displaying the
infoset. This only affects the ``info infoset`` command. This shows the last
number of lines of the infoset. If the number is less than or equal to zero,
the entire infoset is printed. Defaults to ``-1``.
+
+ ``removeHidden <boolean>``
+
+ : Set whether or not hidden elements (e.g through the use of
dfdl:hiddenGroupRef) should be displayed. This effects the ``eval`` and ``info
infoset`` commands. Must be either ``true``/``false`` or ``1``/``0``. Defaults
to ``false``.
+
+ ``representation <string>``
+
+ : Set the output when displaying data. Must be either ``text` or
``binary``. Defaults to ``text``.
+
+ ``wrapLength <integer>``
+
+ : Set the number of characters at which point output wraps. This only
affects the ``info data`` and ``info infoset`` commands. A length less than or
equal to zero disables wrapping. Defaults to ``80``.
+
+
+step
+
+ : Perform a single parse action, pause parsing, and display a debugger
prompt.
+
+ : Abbreviation: ``s``
+
+trace
+
+ : Continue parsing the input data until a breakpoint is encountered, while
running display commands after every parse step. When a breakpoint is
encountered, pause parsing and display a debugger console to the user.
+
+ : Abbreviation: ``t``
diff --git a/site/dfdl-extensions.md b/site/dfdl-extensions.md
new file mode 100644
index 0000000..665df5d
--- /dev/null
+++ b/site/dfdl-extensions.md
@@ -0,0 +1,57 @@
+---
+layout: page
+title: DFDL Extensions
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## DFDL Extensions
+
+Daffodil provides extensions to the DFDL specification. These properties are
in the namespace defined by the URI
``urn:ogf:dfdl:2013:imp:daffodil.apache.org:2018:ext``. To use these the
Daffodil DFDL Extensions, bind this namespace to the ``daf`` prefix in the
``xs:schema`` element, like so:
+
+``` xml
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema";
+ xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/";
+ xmlns:daf="urn:ogf:dfdl:2013:imp:daffodil.apache.org:2018:ext">
+```
+
+The following symbols defined in this namespace are described below.
+
+### Expression Functions
+
+``daf:error()``
+
+ : A function that can be used in DFDL expressions. This functions does not
return a value or accept any arguments. When called, it causes a Parse Error or
Unparse Error.
+
+ *This function is deprecated as of Daffodil 2.0.0. Use the
``fn:error(...)`` function instead.*
+
+``daf:trace($value, $label)``
+
+ : A function that can be used in DFDL expressions, similar to the
``fn:trace()`` function. This logs the string ``$label`` followed by ``$value``
converted to a string and returns ``$value``. The second argument must be of
type ``xs:string``.
+
+### Properties
+
+``daf:parseUnparsePolicy``
+
+ : A property applied to simple and complex elements, which specifies
whether the element supports only parsing, only unparsing, or both parsing and
unparse. Valid values for this property are ``parse``, ``unparse``, or
``both``. This allows one to leave off properties that are required for only
parse or only unparse, such as ``dfdl:outputValueCalc`` or
``dfdl:outputNewLine``, so that one may have a valid schema if only a subset of
functionality is needed.
+
+ All elements must have a compatible parseUnparsePolicy with the
compilation parseUnparsePolicy (which is defined by the root element
daf:parseUnparsePolicy and/or the Daffodil parseUnparsePolicy tunable) or it is
a Schema Definition Error. An element is defined to have a compatible
parseUnparsePolicy if it has the same value as the compilation
parseUnparsePolicy or if it has the value ``both``.
+
+ For compatibility, if this property is not defined, it is assumed to be
``both``.
diff --git a/site/dfdl-layout.md b/site/dfdl-layout.md
new file mode 100644
index 0000000..e75b613
--- /dev/null
+++ b/site/dfdl-layout.md
@@ -0,0 +1,166 @@
+---
+layout: page
+title: Standard DFDL Schema Project Layout
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Standard DFDL Schema Project layout
+
+There is a specific way of organizing a DFDL schema project that has been
found to be helpful. It uses specific directory naming conventions and tree
structure to manage name conflicts in a manner similar to how Java package
names * correspond to directory names. This set of conventions provides a
number of benefits:
+
+* No name conflicts or ambiguity on classpath if multiple DFDL schemas are
used together
+* You can copy multiple DFDL schemas into one directory tree and there will be
no conflicts of file names
+* sbt can be used to
+ * Package the schema into a jar, which can then be on a classpath, become
part of a larger application, etc.
+ * Auto-download all dependencies of the schema, including Daffodil itself.
+ * Run a suite of tests via 'sbt test'
+ * Publish a local version of the schema for use in other projects that
also follow this layout.
+* Eclipse IDE for development and test of the schema. Multiple such schemas
all work together without conflict in the IDE.
+* Encourages organizing DFDL schemas into reusable libraries.
+ * A DFDL schema project need not define a whole data format. It can define
a library of pieces to be included/imported by other formats.
+
+These conventions are actually usable for regular XML-schema projects, that
is, they're not really DFDL-specific conventions. They're general conventions
for organizing projects so as to achieve the above benefits.
+
+### Conventions
+
+We're using Apache (apache.org) here as an example. Substitute your
organization's details. Let's assume the DFDL schema contains two files named
main.dfdl.xsd, and format.dfdl.xsd, and that our format is named RFormat.
+
+The standard file tree would be:
+
+ RFormat/
+ ??? src/
+ ? ??? main/
+ ? ? ??? resources/
+ ? ? ??? org/
+ ? ? ??? apache/
+ ? ? ??? RFormat/
+ ? ? ??? xsd/
+ ? ? ? ??? main.dfdl.xsd - main DFDL schema file
+ ? ? ? ??? format.dfdl.xsd - DFDL schema file
imported/included from main
+ ? ? ??? xsl/
+ ? ? ??? xforms.xsl - resources other than
XSD go in other directories
+ ? ??? test/
+ ? ??? resources/
+ ? ? ??? org/
+ ? ? ??? apache/
+ ? ? ??? RFormat/
+ ? ? ??? tests1.tdml - TDML test files
+ ? ??? scala/
+ ? ??? org/
+ ? ??? apache/
+ ? ??? RFormat/
+ ? ??? Tests1.scala - Scala test driver file
+ ?
+ ??? build.sbt - simple build tool (sbt) specification file. Edit to
change version of Daffodil
+ ? needed, or versions of other DFDL schemas needed
+ ??? README.md - Documentation about the DFDL schema in Markdown file
format
+ ??? .classpath - Eclipse classpath file (optional)
+ ??? .project - Eclipse project file (optional)
+ ??? .gitignore - Git revision control system 'ignore' file (should
contain 'target' and
+ 'lib_managed' entries)
+
+### build.sbt
+
+Use the below template for the build.sbt file:
+
+``` scala
+name := "dfdl-RFormat"
+
+organization := "org.apache"
+
+version := "0.0.1"
+
+scalaVersion := "2.11.8"
+
+crossPaths := false
+
+testOptions in ThisBuild += Tests.Argument(TestFrameworks.JUnit, "-v")
+
+resolvers in ThisBuild ++= Seq(
+ "NCSA Sonatype Releases" at
"https://opensource.ncsa.illinois.edu/nexus/content/repositories/releases";,
+ "NCSA Sonatype Snapshots" at
"https://opensource.ncsa.illinois.edu/nexus/content/repositories/snapshots";
+)
+
+libraryDependencies in ThisBuild := Seq(
+ "edu.illinois.ncsa" %% "daffodil-tdml" % "2.0.0" % "test"
+ "junit" % "junit" % "4.11" % "test",
+ "com.novocode" % "junit-interface" % "0.10" % "test",
+)
+```
+
+Edit the version of daffodil-tdml above to match the version you are using.
+
+### Eclipse IDE
+
+If you organize your DFDL schema project using the above conventions, and then
run ``sbt compile``, the ``lib_managed`` directory will be populated. Then if
you create a new Eclipse scala project from the directory tree, Eclipse will
see the ``lib_managed`` directory and construct a classpath containing all
those jars.
+
+### XSD Conventions
+
+DFDL schemas should have the ``.dfdl.xsd`` suffix to distinguish them from
ordinary XML Schema files.
+
+A DFDL schema should have a target namespace.
+
+Stylistically, the XSD ``elementFormDefault="unqualified"`` is the preferred
style for DFDL schemas.
+
+### Using a DFDL Schema
+
+The ``xs:include`` or ``xs:import`` elements of a DFDL Schema can
import/include a DFDL schema that follows these conventions like this:
+
+``` xml
+<xs:import namespace="urn:tresys.com/RFormat"
schemaLocation="com/tresys/RFormat/xsd/main.dfdl.xsd"/>
+```
+
+The above is for using a DFDL schema as a library, from another different DFDL
schema.
+
+Within a DFDL schema, one DFDL schema file can reference another peer file
that appears in the same directory (the src/main/resources/.../xsd directory)
via:
+
+``` xml
+<xs:include schemaLocation="format.dfdl.xsd"/>
+```
+
+That is, peer files need not carry the long ``com/tresys/RFormat/xsd/`` prefix
that makes the reference globally unique.
+
+However, if one schema wants to include another different schema, then this
standard way of organizing schema projects ensures that when packaged into jar
files, the ``/src/main/resources`` directory contents are at the "root" of the
jar file so that the ``schemaLocation`` of the ``xs:import`` or ``xs:include``
containing the fully qualified path (``com/tresys/RFormat/xsd/main.dfdl.xsd``)
will be found on the ``CLASSPATH`` unambiguously. This convention is what
allows the schema files themselves to have short names like main.dfdl.xsd, and
format.dfdl.xsd. Those names only need to be unique within a single schema
project. Across schema projects our standard DFDL schema project layout insures
unambiguous qualification is available.
+
+### Git Revision Control
+
+You don't have to use Git version control, but many people do, and github.com
is one of the reasons for this popularity.
+
+Each DFDL schema should have its own Git repository if it is going to be
revised independently. We encourage users to join the
[DFDLSchemas](http://dfdlschemas.github.io/) project on github and create
repositories and publish schemas for any publicly-available formats there. For
other formats that are not publicly available, one may want to put a
placeholder for them on DFDLSchemas anyway (as IBM has done for some formats
like Swift-MT.)
+
+### Jar File Packaging
+
+A DFDL schema using the recommended file structure as described here, can be
packaged into a jar for convenient import/include from other schemas.
+
+The sbt command does all the work:
+
+ sbt package # creates jar
+ sbt publishLocal # puts it into local .ivy2 cache where other maven/sbt
will find it.
+
+The resulting jar has the src/main/resources directory in it at the root of
the jar. If this jar is on the classpath, then other schemas containing XSD
import or include statements will search the jar with the schema location.
+
+That enables a different schema's ``build.sbt`` to contain a library
dependency on our hypthetical dfdl-RFormat schema using a dependency like this:
+
+```
+"org.apache" % "dfdl-RFormat" % "0.0.1" % "test"
+```
+
+That will result in the contents of the ``src/main/resources`` directory above
being on the classpath. XSD include and import statements search the classpath
directories.
diff --git a/site/eclipse-configuration.md b/site/eclipse-configuration.md
new file mode 100644
index 0000000..97c43cc
--- /dev/null
+++ b/site/eclipse-configuration.md
@@ -0,0 +1,110 @@
+---
+layout: page
+title: Eclipse Configuration
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Eclipse Configuration for DFDL Schema Authoring/Editing
+
+The Eclipse IDE can be used to assist a user in the authoring of DFDL schemas.
This page contains instructions for setting up the Eclipse
Integrated-Development-Environment (IDE), and to configure its XML editing
features so that they provide some level of support for authoring DFDL schemas.
+
+DFDL extends a subset of XML Schema. Note however: an XML Schema is an XML
document.
+
+Now, turns out there is an XML Schema for DFDL Schemas. It was created by
taking the XML Schema for XML Schemas, and subsetting it to just what DFDL
uses, and then adding into it the XML Schema for DFDL's annotation objects.
+
+### Installing Eclipse XML Plugin
+
+The XML settings all assume you have installed Eclipse's XML support. If not
already installed, the "Eclipse XML Editors and Tools" plugin can be Eclipse's
update site.
+
+### Eclipse XML Settings
+
+Following are steps to modify the Eclipse XML settings to aid in the creation
of DFDL schemas:
+
+#### Enable Validation of DFDL Files
+
+1. Window > Preferences > Validation
+ 1. Turn off XML Schema Validator
+ 1. Turn off DTD Validator
+ 1. Click the ... settings box to the right of XML Validator
+ 1. Select Include Group
+ 1. Click Add Rule
+ 1. Select File Extension
+ 1. Click Next
+ 1. Add extension: ``dfdl.xsd``
+ 1. Click Finish
+1. Window > Preferences > General > Content Types
+ 1. Select Text > XML
+ 1. Click Add...
+ 1. Set Content type to ``*.dfdl.xsd``
+1. Window > Preferences > XML > XML Files > Editor
+ 1. Uncheck Format Comments
+
+If also editing TDML file in Eclipse, repeat the above steps, replacing all
instances of ``dfdl.xsd`` with ``tdml``.
+
+#### Add XML Schemas for DFDL to the XML Catalog
+
+Download the following files to a local directory. Alternatively, these files
are included in the Daffodil source code in the
``src/main/resources/edu/illinois/ncsa/daffodil/xsd/`` directory in either
``daffodil-lib/`` or ``daffodil-propgen/``.
+
+*
[datatypes.dtd](https://raw.githubusercontent.com/apache/incubator-daffodil/master/daffodil-lib/src/main/resources/edu/illinois/ncsa/daffodil/xsd/datatypes.dtd)
+*
[tdml.xsd](https://raw.githubusercontent.com/apache/incubator-daffodil/master/daffodil-lib/src/main/resources/edu/illinois/ncsa/daffodil/xsd/tdml.xsd)
+*
[XMLSchema.dtd](https://raw.githubusercontent.com/apache/incubator-daffodil/master/daffodil-lib/src/main/resources/edu/illinois/ncsa/daffodil/xsd/XMLSchema.dtd)
+*
[XMLSchema_for_DFDL.xsd](https://raw.githubusercontent.com/apache/incubator-daffodil/master/daffodil-lib/src/main/resources/edu/illinois/ncsa/daffodil/xsd/XMLSchema_for_DFDL.xsd)
+*
[DFDL_part1_simpletypes.xsd](https://raw.githubusercontent.com/apache/incubator-daffodil/master/daffodil-propgen/src/main/resources/edu/illinois/ncsa/daffodil/xsd/DFDL_part1_simpletypes.xsd)
+*
[DFDL_part2_attributes.xsd](https://raw.githubusercontent.com/apache/incubator-daffodil/master/daffodil-propgen/src/main/resources/edu/illinois/ncsa/daffodil/xsd/DFDL_part2_attributes.xsd)
+*
[DFDL_part3_model.xsd](https://raw.githubusercontent.com/apache/incubator-daffodil/master/daffodil-propgen/src/main/resources/edu/illinois/ncsa/daffodil/xsd/DFDL_part3_model.xsd)
+
+Modify the Eclipse settings to reference these files:
+
+1. Windows > Preferences > XML > XML Catalog
+1. For each of the following files, select 'Add..' and provide the location on
the 'File System...' or 'Workspace...', along with the key and key type
+
+ <table class="table">
+ <tr>
+ <th>File</th>
+ <th>Key Type</th>
+ <th>Key</th>
+ </tr>
+ <tr>
+ <td>XMLSchema.dtd</td>
+ <td>Public ID</td>
+ <td>-//W3C//DTD XMLSCHEMA 200102//EN</td>
+ </tr>
+ <tr>
+ <td>datatypes.dtd</td>
+ <td>Public ID</td>
+ <td>datatypes</td>
+ </tr>
+ <tr>
+ <td>XMLSchema_for_DFDL.xsd</td>
+ <td>URI</td>
+ <td>http://www.w3.org/2001/XMLSchema</td>
+ </tr>
+ <tr>
+ <td>DFDL_part3_model.xsd</td>
+ <td>URI</td>
+ <td>http://www.ogf.org/dfdl/dfdl-1.0/</td>
+ </tr>
+ <tr>
+ <td>tdml.xsd</td>
+ <td>URI</td>
+ <td>http://www.ibm.com/xmlns/dfdl/testData</td>
+ </tr>
+ </table>
diff --git a/site/examples.md b/site/examples.md
new file mode 100644
index 0000000..694fd56
--- /dev/null
+++ b/site/examples.md
@@ -0,0 +1,158 @@
+---
+layout: page
+title: Examples
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+
+## Examples
+
+Below are two examples of how Daffodil parses Comma-Separated Values (CSV) and
Packet Capture data (PCAP) data to XML. For simplicity, each example uses the
Daffodil [Command Line Interface](/cli) to parse input data to output XML.
+
+### Comma-Separated Values (CSV)
+
+This DFDL schema is found at the DFDLSchemas GitHub [CSV
repository](https://github.com/DFDLSchemas/CSV).
+
+```bash
+$ daffodil parse --schema
CSV/src/main/resources/com/tresys/csv/xsd/csv.dfdl.xsd
CSV/src/test/resources/com/tresys/csv/data/simpleCSV.csv
+```
+The above command uses the csv.dfdl.xsd schema to parse the simpleCSV.csv
input data. The simpleCSV.csv contains the following data:
+
+```
+last,first,middle,DOB
+smith,robert,brandon,1988-03-24
+johnson,john,henry,1986-01-23
+jones,arya,cat,1986-02-19
+```
+
+The result of the parse is the following XML:
+
+```xml
+<ex:file xmlns:ex="http://example.com";>
+ <header>
+ <title>last</title>
+ <title>first</title>
+ <title>middle</title>
+ <title>DOB</title>
+ </header>
+ <record>
+ <item>smith</item>
+ <item>robert</item>
+ <item>brandon</item>
+ <item>1988-03-24</item>
+ </record>
+ <record>
+ <item>johnson</item>
+ <item>john</item>
+ <item>henry</item>
+ <item>1986-01-23</item>
+ </record>
+ <record>
+ <item>jones</item>
+ <item>arya</item>
+ <item>cat</item>
+ <item>1986-02-19</item>
+ </record>
+</ex:file>
+```
+
+### Packet Capture Data (PCAP)
+
+This DFDL schema is found in the DFDLSchemas GitHub [PCAP
repository](https://github.com/DFDLSchemas/PCAP).
+
+The PCAP file format is a binary file format used to capture network packets.
For information on this file format, visit the [Wireshark Libpcap File
Format](http://wiki.wireshark.org/Development/LibpcapFileFormat) page.
+
+```bash
+$ daffodil parse --schema
PCAP/src/main/resources/com/tresys/pcap/xsd/pcap.dfdl.xsd
PCAP/src/test/resources/com/tresys/pcap/data/icmp.cap
+```
+
+The above command uses the pcap.dfdl.xsd schema to parse the icmp.cap input
file. Viewed as a hex dump, the icmp.cap file looks like:
+
+```
+0000000 c3d4 a1b2 0002 0004 0000 0000 0000 0000
+0000020 ffff 0000 0001 0000 6fc4 51c1 ccf8 000c
+0000040 004a 0000 004a 0000 5000 e056 4914 0c00
+...
+0001300 0000 5c2f 0002 0024 6261 6463 6665 6867
+0001320 6a69 6c6b 6e6d 706f 7271 7473 7675 6177
+0001340 6362 6564 6766 6968
+```
+
+The result of the parse is the following XML:
+
+```xml
+<pcap:PCAP xmlns:pcap="urn:pcap:2.4">
+ <PCAPHeader>
+ <MagicNumber>D4C3B2A1</MagicNumber>
+ <Version>
+ <Major>2</Major>
+ <Minor>4</Minor>
+ </Version>
+ <Zone>0</Zone>
+ <SigFigs>0</SigFigs>
+ <SnapLen>65535</SnapLen>
+ <Network>1</Network>
+ </PCAPHeader>
+ <Packet>
+ <PacketHeader>
+ <Seconds>1371631556</Seconds>
+ <USeconds>838904</USeconds>
+ <InclLen>74</InclLen>
+ <OrigLen>74</OrigLen>
+ </PacketHeader>
+ <pcap:LinkLayer>
+ <pcap:Ethernet>
+ <MACDest>005056E01449</MACDest>
+ <MACSrc>000C29340BDE</MACSrc>
+ <Ethertype>2048</Ethertype>
+ <pcap:NetworkLayer>
+ <pcap:IPv4>
+ <IPv4Header>
+ <Version>4</Version>
+ <IHL>5</IHL>
+ <DSCP>0</DSCP>
+ <ECN>0</ECN>
+ <Length>60</Length>
+ <Identification>55107</Identification>
+ <Flags>0</Flags>
+ <FragmentOffset>0</FragmentOffset>
+ <TTL>128</TTL>
+ <Protocol>1</Protocol>
+ <Checksum>11123</Checksum>
+ <IPSrc>192.168.158.139</IPSrc>
+ <IPDest>174.137.42.77</IPDest>
+ </IPv4Header>
+ <PayloadLength>40</PayloadLength>
+ <Protocol>1</Protocol>
+ <pcap:ICMPv4>
+ <Type>8</Type>
+ <Code>0</Code>
+ <Checksum>10844</Checksum>
+ <Data>02002100</Data>
+ </pcap:ICMPv4>
+ </pcap:IPv4>
+ </pcap:NetworkLayer>
+ </pcap:Ethernet>
+ </pcap:LinkLayer>
+ </Packet>
+ ...
+</pcap:PCAP>
+```
diff --git a/site/faq.md b/site/faq.md
new file mode 100644
index 0000000..4203fab
--- /dev/null
+++ b/site/faq.md
@@ -0,0 +1,291 @@
+---
+layout: page
+title: Frequently Asked Questions
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style type="text/css">
+.in, .collapsing {
+ margin-left: 20px;
+}
+.question {
+ cursor: pointer;
+}
+</style>
+
+## Frequently Asked Questions
+
+{% assign faqNum=faqNum | plus:'1' %}
+<a class="question" data-toggle="collapse" data-target="#faq{{faqNum}}">
+ Q: When should I use an XSD facet like maxLength, and when should I use the
DFDL length property?
+</a>
+<div id="faq{{faqNum}}" class="collapse" markdown="1">
+Here's part of an example from the DFDL tutorial of a street address:
+
+``` xml
+<xs:element name="houseNumber" type="xs:string" dfdl:lengthKind="explicit"
dfdl:length="6"/>
+```
+
+Note that the length of the house number is constrained with DFDL. XSD can
also be used to constrain lengths.
+
+When should you used XSD to do this, and when should you use DFDL? Should you
ever use both?
+
+You must use the dfdl:length property, because it can't parse the data without
it. You may use the XSD facets to check further, and it often makes sense to
use both.
+
+Consider
+
+``` xml
+<xs:element name="article" type="xs:string" dfdl:length="{
../header/articleLength }" dfdl:lengthKind='explicit'/>
+```
+
+Now the length is coming from a field someplace at runtime. Validating that it
is within some additional constraints on maxLength might be very valuable. To
do that you nave to write the more verbose:
+
+``` xml
+<xs:element name="article" dfdl:length="{ ../header/articleLength }"
dfdl:lengthKind='explicit'>
+ <xs:simpleType>
+ <xs:restriction base="xs:string">
+ <xs:maxLength value="140"/>
+ </xs:restriction>
+ </xs:simpleType>
+</xs:element>
+```
+
+Not too bad actually. And if you can reuse some simple type definitions it's
not bad at all.
+
+One further point. Suppose you want to parse the string using the
header-supplied length, but it's flat out a parse error if the length turns out
to be greater than 140. You can ask the DFDL processor to check the facet
maxLength at parse time using an assertion like this:
+
+``` xml
+<xs:element name="article" dfdl:length="{ ../header/articleLength }"
dfdl:lengthKind='explicit'>
+ <xs:simpleType>
+ <xs:annotation><xs:appinfo source="http://www.ogf.org/dfdl/dfdl-1.0";>
+ <dfdl:assert>{ dfdl:checkConstraints() }</dfdl:assert>
+ </xs:appinfo></xs:annotation>
+ <xs:restriction base="xs:string">
+ <xs:maxLength value="140"/>
+ </xs:restriction>
+ </xs:simpleType>
+</xs:element>
+```
+
+The dfdl:assert statement annotation calls a built-in DFDL function called
dfdl:checkConstraints, which tells DFDL to test the facet constraints and issue
a parse error if they are not satisfied. This is particularly useful for
enumeration constraints where an element value is an identifier of some sort.
+</div>
+
+
+
+
+{% assign faqNum=faqNum | plus:'1' %}
+<a class="question" data-toggle="collapse" data-target="#faq{{faqNum}}">
+ Q: Should I use dfdl:assert to validate while parsing?
+</a>
+<div id="faq{{faqNum}}" class="collapse" markdown="1">
+In general, no. The dfdl:assert statement annotation should be used to guide
the parser. It should test things that must be true in order to successfully
parse the data and create an Infoset from it.
+
+But, it should not be used to ensure validation of the values of the data
elements.
+
+By way of illustrating what not to do, it is tempting to put facet constraints
on simple type definitions in your schema, and then use a dfdl:assert like this:
+
+```xml
+<dfdl:assert>{ checkConstraints(.) }</dfdl:assert>
+```
+
+so that the parser will validate as it parses, and will fail to parse values
that do not satisfy the facet constraints.
+
+Don't do this. Your schema will not be as useful because it will not be able
to be used for some applications, for example, applications that want to accept
well-formed, but invalid data and analyze, act, or report on the invalid
aspects.
+
+In some sense, embedding checks like this into a DFDL schema is
second-guessing the application's needs, and assuming the application does not
even want to successfully parse and create an infoset from data that does not
obey the facet constraints.
+</div>
+
+
+
+
+
+{% assign faqNum=faqNum | plus:'1' %}
+<a class="question" data-toggle="collapse" data-target="#faq{{faqNum}}">
+ Q: How do I prevent my DFDL expressions and regular expressions from being
modified by my XML editor?
+</a>
+<div id="faq{{faqNum}}" class="collapse" markdown="1">
+Use CDATA with expressions and regular expressions, and generally to stop XML
editors from messing with your DFDL schema layouts.
+
+Most XML editors will wrap long lines. So your
+
+``` xml
+<a>foobar</a>
+```
+
+just might get turned into
+
+``` xml
+<a>foobar
+</a>
+```
+
+Now most of the time that is fine. But sometimes the whitespace really
matters. One such place is when you type a regular expression. In DFDL this can
come up in this way:
+
+``` xml
+<dfdl:assert testKind="pattern"> *</dfdl:assert>
+```
+
+Now the contents of that element is " \*", i.e., a single space, and the "\*"
character. That means zero or more spaces in regex language. If you don't want
your XML tooling to mess with the whitespace do this instead:
+
+``` xml
+<dfdl:assert testKind="pattern"><![CDATA[ *]]></dfdl:assert>
+```
+
+CDATA informs XML processors that you very much care about this. Any decent
XML tooling/editor will see this and decide it cannot line-wrap this or in any
way mess with the whitespace. Also useful if you want to write a complex DFDL
expression in the expression language, and you want indentation and lines to be
respected. Here's an example:
+
+``` xml
+<dfdl:discriminator><![CDATA[{
+ if (daf:trace((daf:trace(../../ex:presenceBit,"presenceBit") =
0),"pbIsZero")) then false()
+ else if
+ (daf:trace(daf:trace(dfdl:occursIndex(),"occursIndex") = 1,"indexIsOne"))
then true()
+ else if
+
(daf:trace(daf:trace(xs:int(daf:trace(../../ex:A1[daf:trace(dfdl:occursIndex()-1,"indexMinusOne")],
+
"occursIndexMinusOneNode")/ex:repeatBit),
+ "priorRepeatBit") = 0,
+ "priorRepeatBitIsZero"))
+ then false()
+ else true()
+}]]></dfdl:discriminator>
+```
+
+If you get done writing something very deeply nested like this (and XPath
style languages require this all the time), then you do NOT want anything
messing with the whitespace.
+
+About the xml:space='preserve' attribute: According to this thread on the
stack overflow web site, xml:space is only about whitespace-only nodes, not
nodes that are part whitespace. Within element-only content, the text nodes
found between the elements are whitespace-only nodes. Unless you use
xml:space='preserve', those are eliminated. None of the above discussion is
about whitespace-only nodes. It's about value nodes containing text strings
with surrounding whitespace.
+</div>
+
+
+
+{% assign faqNum=faqNum | plus:'1' %}
+<a class="question" data-toggle="collapse" data-target="#faq{{faqNum}}">
+ Q: Why doesn't DFDL allow me to express my format using offsets into a file,
instead of lengths?
+</a>
+<div id="faq{{faqNum}}" class="collapse" markdown="1">
+With some study, the DFDL workgroup concluded that these formats nearly always
require the full complexity of a transformation system AND a data format
description system. DFDL is only about the latter problem.
+
+In other words, it was left out for complexity reasons, not because we didn't
think there were examples.
+
+It is a much more complex issue than people think. As we got into it we kept
falling down the slippery slope of needing rich transformations to express such
things.
+
+We certainly have seen formats where there are a bunch of fields, in the
ordinary manner, but instead of expressing their lengths, the forrmat specifies
only their starting positions relative to start of record. There are also
formats where there are tables of offsets into a subsequent data array.
+
+DFDL requires one to recast such a specification as lengths.
+
+It is not a "either or" scenario where lengths and offsets are equivalent so
you can pick one.
+
+Use of lengths is simply a superior and more precise way of expressing the
format because use of offsets can obscure aliasing, which is the term for when
there are two elements (or more) that describe the same part of the data
representation. With lengths, it's clear what every bit means, and that every
bit is in fact described or explicitly skipped. You can't just use an offset to
skip past a bunch of data leaving it not described at all. You can't have
aliasing of the same data.
+
+Aliasing is a difficult issue when parsing. When unparsing it is a nightmare,
as it introduces non-determinacy in what the data written actually comes out
like. It depends on who writes it last with what alias.
+
+Structures like
+
+ <offset to start><length of thing>
+ <offset to start2><length of thing2>
+ ...
+ <offset to startN><length of thingN>
+ thing
+ thing2
+ ...
+ thingN
+
+So long as the things and the corresponding descriptor pairs are in order,
these can be described. The lengths need not even be there as they are
redundant. If present they can be checked for validity. Overlap can be checked
for and deemed invalid.
+
+But, in DFDL the above *must* be represented as two vectors. One of the
offsets table, the other of the things. If you want an array of things and then
want DFDL to convert that into the offsets and things separately, well DFDL
doesn't do transformations of that sort. Do that first in XSLT or other
transformation system when unparsing. When parsing, you first parse with DFDL,
then transform the data into the logical single vector using XSLT (or other).
+
+<div class="alert alert-warning">
+XProc is a language for expressing chains of XML-oriented transformations like
this. Calabash is an open-source XProc implementation, and the
daffodil-calabash-extension provides Daffodil stages that have been created to
enable creation of XProc pipelines that glue together transformations like XSLT
with DFDL parse/unparse steps. This can be used to create a unit that runs both
DFDL and an XSLT together for parse or for unparse (they would be different
XSLTs).
+If the things are potentially out of order, especially if the lengths are not
stored, but just implied by "from this offset to the start of the next one,
whichever one that is", that is simply too complex a transformation for DFDL.
+</div>
+
+If you think about what is required mentally to decode this efficiently, you
must grab all the entries, sort them by offset, and then compute lengths, etc.
Shy of building a real programming language (e.g., XQuery) into DFDL there has
to be a limit to what level of complexity we allow DFDL to express directly.
And unparsing is entirely non-deterministic... you have to stage an array/blob
filled with fill bytes, write pieces to it one by one, potentially overwriting
sections. It's really quite hard. Even if you supported this in DFDL somehow,
would it in fact write these things out in the order an application does? So
will you even be able to re-create data?
+
+There is a sense in which formats expressed as these sorts of "potentially
overlapping regions" are simply not adequately specified unless they specify
the exact order things are to be written so that the contents of overlap
regions is deterministic.
+
+There could be formats where there are offset tables like this, where in
principle things could be out of order, or overlapping/aliased, but they simply
never are, and allowing them to be is effectively a bad idea as it allows
people to do very obscure things - information hiding, polyglot files, etc. PDF
is heavily criticized for this. It may be an unstated principle that such
formats *do not* do this sort of out-of-order or aliasing stuff.
+
+All that said, practically speaking, people have data with offset tables, and
out-of-order might be a possibility that needs to be allowed at least on
parsing. So what to do in DFDL?
+
+In this case, DFDL can describe the table of offsets, and a big blob of data.
Beyond that something else (e.g., XSLT, or a program) must take over for
expressing the sort and extraction of chunks out of the larger blob.
+
+If you think about this, if you want deterministic unparsing behavior, that is
what has to be presented to the DFDL unparser anyway, since presenting the
resolved content blob means the application has dealt with the order to which
the various chunks (which may overlap) have been written.
+</div>
+
+
+{% assign faqNum=faqNum | plus:'1' %}
+<a class="question" data-toggle="collapse" data-target="#faq{{faqNum}}">
+ Q: How can I get strings in the data to become element names?
+</a>
+<div id="faq{{faqNum}}" class="collapse" markdown="1">
+If the data contains tags/strings, and you want those strings to become
element names in XML, then you *must* do pass 1 to extract the tag information,
use them as element names when you create a DFDL schema dynamically, and then
parse the data again with this new specialized DFDL schema.
+
+Or you can parse the data with a generic schema where your tag names will be
in element values someplace, and do a transformation outside of DFDL to convert
them to element names.
+
+Consider the common "comma separated values" or CSV formats. If you have
+
+```
+Name, Address, Phone
+Mike, 8840 Standford Blvd\, Columbia MD, 888-888-8888
+```
+
+and you want
+
+``` xml
+<columnNames>
+ <name>Name</name>
+ <name>Address</name>
+ <name>Phone</name>
+</columnNames>
+<row>
+ <col>Mike</col>
+ <col>8840 Standford Blvd, Columbia MD</col>
+ <col>888-888-8888</col>
+</row>
+```
+That's what you would get from a generic CSV DFDL schema. If you want this:
+
+``` xml
+<row>
+ <Name>Mike</Name>
+ <Address>8840 Stanford Blvd, Columbia MD</Address>
+ <Phone>888-888-8888</Phone>
+</row>
+```
+
+That's a specific-to-exactly-these-column-names CSV DFDL schema that is
required. If you have lots of files with this exact structure you would create
this DFDL schema once.
+
+If you have no idea what CSV is coming at you, but want this sort of XML
elements anyway, then you have to generate a DFDL schema on the fly from the
data (parse just the headers with a generic DFDL schema first - then use that
to create the DFDL schema.
+
+Or you parse using the generic schema, then use XSLT or something to convert
the result of the generic parse.
+
+Keep in mind that this problem has little to do with DFDL. Given an XML
document like the generic one above, but you didn't want that XML, you wanted
the specific style XML. Well you have the same problem. You need to grab the
column names first, then transform the data using them as the element names.
+</div>
+
+<!---
+Copy the below snippet and edit to create a new FAQ entry, changing the
+question and answer only. Everything else should remain exactly the same.
+
+{% assign faqNum=faqNum | plus:'1' %}
+<a class="question" data-toggle="collapse" data-target="#faq{{faqNum}}">
+ Q: Question Goes Here
+</a>
+<div id="faq{{faqNum}}" class="collapse" markdown="1">
+ Answer goes here using markdown
+</div>
+-->
diff --git a/site/getting-started.md b/site/getting-started.md
new file mode 100644
index 0000000..e46a060
--- /dev/null
+++ b/site/getting-started.md
@@ -0,0 +1,67 @@
+---
+layout: page
+title: Getting Started
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+## Getting Started
+
+### Using Daffodil
+
+Daffodil is a library, requiring Java 8, used to convert between fixed format
data and XML/JSON based on a DFDL schema. Some [examples](/examples) show the
result of Daffodil parsing various inputs into XML. Multiple mechanisms that
use the Daffodil library invoke its capabilities are described below.
+
+[Command Line Interface](/cli)
+
+ : Linux and Windows command line tool with capabilities include parsing,
unaparsing, an interactive debugger, and more. Available via download in binary
[releases](/releases).
+
+[Java API](/docs/latest/javadoc) or [Scala API](/docs/latest/scaladoc)
+
+ : Examples for using the Java API are available on the [OpenDFDL
examples](https://github.com/OpenDFDL/examples.git) repository. Individual
[releases](/releases) describe how to include a dependency to Daffodil via
Maven and SBT.
+
+[Apache NiFi
Processors](https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil-nifi/browse)
+
+ : Provides two [Apache NiFi](https://nifi.apache.org/) processors for
parsing and unparsing NiFi FlowFiles.
+
+[XML Calabash
Extension](https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil-nifi/browse)
+
+ : An extension to [XML Calabash](http://xmlcalabash.com) that declares
XProc pipeline steps to parse and unparse input data.
+
+### Existing DFDL Schemas
+
+Many DFDL schemas are freely and publicly available at
[DFDLSchemas](https://github.com/DFDLSchemas) on GitHub. This includes image
formats, financial transaction formats, healthcare formats, and more. Some DFDL
Schemas for Controlled Unclassified Information (CUI) data formats are
available on
[DI2E.net](https://confluence.di2e.net/pages/viewpage.action?pageId=196139975),
which requires US Citizenship and DoD sponsorship.
+
+
+### Creating DFDL Schemas
+
+#### Layout
+
+When creating a custom DFDL schema, it is beneficial to follow the
directory/file layout described in [Standard DFDL Schema Project
Layout](/dfdl-layout). This layout provides a consistent and familiar structure
for DFDL schema development that existing tooling understands, provides for
easy testing and schema packaging, and ensures no name conflicts on classpaths
if multiple schemas are used together.
+
+#### Editors
+
+Any text editor can be used for authoring DFDL schemas. But because DFDL
schemas are a subset of XML Schema, using an XML editor (especially an XML
Schema aware editor) can provide intelligent assistance in authoring DFDL
Schemas, including attribute and tag completion, validation, indentation,
visualizations, etc. Below are resources for configuring various tools
specifically for editing or authoring DFDL schemas.
+
+ * [Eclipse IDE](/eclipse-configuration)
+
+#### DFDL Extensions
+
+Daffodil provides extensions to the DFDL specification to add extra
enhancements and capabilities. These extensions are listed at [DFDL
Extensions](/dfdl-extensions).
+
diff --git a/site/people.md b/site/people.md
index d8abb89..aa9fa5c 100644
--- a/site/people.md
+++ b/site/people.md
@@ -23,7 +23,7 @@ limitations under the License.
{% endcomment %}
-->
-### People
+## People
{% if site.data.contributors %}
<table class="table">
diff --git a/site/unsupported.md b/site/unsupported.md
new file mode 100644
index 0000000..ff7fe36
--- /dev/null
+++ b/site/unsupported.md
@@ -0,0 +1,232 @@
+---
+layout: page
+title: Unsupported Features and Errata
+group: nav-right
+---
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+
+<style type='text/css'>
+th {
+ background-color: #f4f4f4;
+}
+
+td, th {
+ padding: 5px 10px;
+ border-top: solid 1px #cccccc;
+ width: 20px;
+}
+
+table {
+ display:table;
+ width:100%;
+}
+
+th:last-child {
+ width:100%;
+}
+</style>
+
+## Unsupported Features & Errata
+
+The following table lists unsupported features and errata of the DFDL
Specification since Daffodil 1.0.0 and if/when those features were implemented.
Any features/errata not listed in this table were implemented prior to 1.0.0.
+
+{% ok %} = supported   {% err %} = unsupported   {% warn %} =
partially supported
+
+| 1.0.0 | 1.1.0 | 2.0.0 |Unparsing
+|:--------:|:--------:|:--------:|----------
+|{% err %}|{% warn %}|{% ok %}|unparser
+|{% err %}|{% ok %}|{% ok %}|outputNewLine
+|{% err %}|{% err %}|{% ok %}|fillByte
+|{% err %}|{% ok %}|{% ok %}|textPadKind
+|{% err %}|{% err %}|{% ok %}|textOutputMinLength
+|{% err %}|{% ok %}|{% ok %}|generateEscapeBlock
+|{% err %}|{% ok %}|{% ok %}|extraEscapedCharacters
+|{% err %}|{% err %}|{% ok %}|truncateSpecifiedLengthString
+|{% err %}|{% err %}|{% ok %}|outputValueCalc
+
+
+| 1.0.0 | 1.1.0 | 2.0.0 |Types
+|:--------:|:--------:|:--------:|----------
+|{% ok %}|{% ok %}|{% ok %}|decimal
+|{% ok %}|{% ok %}|{% ok %}|integer
+|{% ok %}|{% ok %}|{% ok %}|nonNegativeInteger
+|{% ok %}|{% ok %}|{% ok %}|hexBinary
+|{% err %}|{% err %}|{% ok %}|boolean
+|{% err %}|{% err %}|{% err %}|date (binary)
+|{% err %}|{% err %}|{% err %}|time (binary)
+|{% err %}|{% err %}|{% err %}|dateTime (binary)
+
+
+| 1.0.0 | 1.1.0 | 2.0.0 |XML Schema Features
+|:--------:|:--------:|:--------:|----------
+|{% err %}|{% err %}|{% err %}|fixed
+|{% err %}|{% err %}|{% err %}|default
+|{% ok %}|{% ok %}|{% ok %}|elementFormDefault
+
+
+| 1.0.0 | 1.1.0 | 2.0.0 |DFDL Features
+|:--------:|:--------:|:--------:|----------
+|{% err %}|{% err %}|{% err %}|defaulting of values for required elements
+|{% err %}|{% err %}|{% err %}|'packed' binary number representations
(packed, bcd, etc.)
+|{% err %}|{% err %}|{% err %}|'zoned' binary number representations
+|{% ok %}|{% ok %}|{% ok %}|advanced text number format properties
+|{% err %}|{% err %}|{% err %}|bi-directional text
+|{% err %}|{% err %}|{% err %}|unordered sequences
+|{% err %}|{% err %}|{% err %}|floating elements in sequences
+|{% ok %}|{% ok %}|{% ok %}|external variables
+|{% err %}|{% err %}|{% err %}|byte-value entities (aka raw-byte entities)
+|{% ok %}|{% ok %}|{% ok %}|validation
+|{% ok %}|{% ok %}|{% ok %}|alignment
+|{% err %}|{% err %}|{% err %}|'type' component of DFDL infoset
+|{% err %}|{% err %}|{% err %}|'unionMemberSchema' component of DFDL infoset
+|{% err %}|{% err %}|{% err %}|'valid' component of DFDL infoset
+|{% err %}|{% err %}|{% err %}|unicodeByteOrderMark
+|{% err %}|{% err %}|{% ok %}|all runtime-computed format properties
+|{% ok %}|{% ok %}|{% ok %}|runtime-computation for escapeCharacter,
escapeEscapeCharacter, byteOrder
+|{% err %}|{% err %}|{% ok %}|IANA standard encodings other than utf-8,
utf-16BE, utf-16LE, utf-32BE, utf-32LE, and ASCII
+|{% ok %}|{% ok %}|{% ok %}|optional qualified names in expression
language
+|{% ok %}|{% ok %}|{% ok %}|check for non-portable regular expressions
+|{% err %}|{% err %}|{% err %}|delimited binary
+|{% err %}|{% err %}|{% err %}|extended ICU symbols 'u' and 'I' in
calendarPattern
+
+
+| 1.0.0 | 1.1.0 | 2.0.0 |Daffodil Specific
+|:--------:|:--------:|:--------:|----------
+|{% ok %}|{% ok %}|{% ok %}|save/reload parser
+
+
+| 1.0.0 | 1.1.0 | 2.0.0 |Properties and Property Enumerations
+|:--------:|:--------:|:--------:|----------
+|{% err %}|{% err %}|{% ok %}|ignoreCase="yes"
+|{% err %}|{% err %}|{% err %}|utf16Width="variable"
+|{% err %}|{% err %}|{% err %}|textNumberRep="zoned"
+|{% err %}|{% err %}|{% err %}|binaryNumberRep="packed"
+|{% err %}|{% err %}|{% err %}|binaryNumberRep="bcd"
+|{% err %}|{% err %}|{% err %}|binaryNumberRep="ibm4690Packed"
+|{% err %}|{% err %}|{% err %}|binaryFloatRep="ibm390Hex"
+|{% err %}|{% err %}|{% err %}|nilKind="logicalValue"
+|{% err %}|{% err %}|{% err %}|nilKind="literalCharacter"
+|{% err %}|{% err %}|{% err %}|lengthKind="prefixed"
+|{% err %}|{% err %}|{% err %}|lengthKind="endOfParent"
+|{% err %}|{% err %}|{% err %}|occursCountKind="stopValue"
+|{% err %}|{% err %}|{% err %}|documentFinalTerminatorCanBeMissing="yes"
+|{% ok %}|{% ok %}|{% ok %}|textNumberPattern
+|{% ok %}|{% ok %}|{% ok %}|textNumberCheckPolicy
+|{% ok %}|{% ok %}|{% ok %}|textNumberRounding
+|{% ok %}|{% ok %}|{% ok %}|textNumberRoundingMode
+|{% ok %}|{% ok %}|{% ok %}|textNumberRoundingIncrement
+|{% ok %}|{% ok %}|{% ok %}|textStandardGroupingSeparator
+|{% ok %}|{% ok %}|{% ok %}|textStandardDecimalSeparator
+|{% ok %}|{% ok %}|{% ok %}|textStandardExponentRep
+|{% ok %}|{% ok %}|{% ok %}|textStandardInfinityRep
+|{% ok %}|{% ok %}|{% ok %}|textStandardNaNRep
+|{% ok %}|{% ok %}|{% ok %}|textStandardZeroRep
+|{% err %}|{% err %}|{% err %}|textStandardBase (behaves as 10)
+|{% err %}|{% err %}|{% err %}|textZonedSignStyle
+|{% err %}|{% err %}|{% ok %}|decimalSigned
+|{% err %}|{% err %}|{% ok %}|textBooleanTrueRep
+|{% err %}|{% err %}|{% ok %}|textBooleanFalseRep
+|{% err %}|{% err %}|{% ok %}|textBooleanPadCharacter
+|{% err %}|{% err %}|{% ok %}|textBooleanJustification
+|{% err %}|{% err %}|{% ok %}|binaryBooleanTrueRep
+|{% err %}|{% err %}|{% ok %}|binaryBooleanFalseRep
+|{% err %}|{% err %}|{% err %}|textBidi (behaves as "no")
+|{% err %}|{% err %}|{% err %}|textBidiTextOrdering
+|{% err %}|{% err %}|{% err %}|textBidiOrientation
+|{% err %}|{% err %}|{% err %}|textBidiSymmetric
+|{% err %}|{% err %}|{% err %}|textBidiTextShaped
+|{% err %}|{% err %}|{% err %}|textBidiNumeralShapes
+|{% ok %}|{% ok %}|{% ok %}|alignment
+|{% ok %}|{% ok %}|{% ok %}|alignmentUnits
+|{% ok %}|{% ok %}|{% ok %}|leadingSkip
+|{% ok %}|{% ok %}|{% ok %}|trailingSkip
+|{% err %}|{% err %}|{% err %}|prefixIncludesPrefixLength
+|{% err %}|{% err %}|{% err %}|prefixLengthType
+|{% ok %}|{% ok %}|{% ok %}|binaryDecimalVirtualPoint
+|{% err %}|{% err %}|{% err %}|binaryNumberCheckPolicy
+|{% err %}|{% err %}|{% err %}|binaryPackedSignCodes
+|{% err %}|{% err %}|{% err %}|binaryCalendarRep
+|{% err %}|{% err %}|{% err %}|binaryCalendarEpoch
+|{% err %}|{% err %}|{% err %}|useNilForDefault
+|{% err %}|{% err %}|{% err %}|sequenceKind='unordered'
+|{% err %}|{% err %}|{% err %}|floating="yes"
+|{% err %}|{% err %}|{% err %}|choiceLengthKind="explicit"
+|{% err %}|{% err %}|{% err %}|choiceLength
+|{% err %}|{% err %}|{% err %}|occursStopValue
+|{% ok %}|{% ok %}|{% ok %}|seperatorSuppressionPolicy (overrides
separatorPolicy)
+|{% ok %}|{% ok %}|{% ok %}|separatorPolicy
+|{% ok %}|{% ok %}|{% ok %}|bitOrder
+|{% ok %}|{% ok %}|{% ok %}|encodingErrorPolicy="replace"
+|{% err %}|{% err %}|{% err %}|encodingErrorPolicy="error"
+|{% err %}|{% err %}|{% err %}|failureType="recoverableError"
+|{% err %}|{% err %}|{% ok %}|choiceDispatchKey
+|{% err %}|{% err %}|{% ok %}|choiceBranchKey
+
+| 1.0.0 | 1.1.0 | 2.0.0 |Functions
+|:--------:|:--------:|:--------:|----------
+|{% err %}|{% err %}|{% ok %}|contentLength
+|{% err %}|{% err %}|{% ok %}|valueLength
+
+| 1.0.0 | 1.1.0 | 2.0.0 |Annotations
+|:--------:|:--------:|:--------:|----------
+|{% err %}|{% err %}|{% err %}|newVariableInstance
+
+| 1.0.0 | 1.1.0 | 2.0.0 |Miscellaneous
+|:--------:|:--------:|:--------:|----------
+|{% err %}|{% err %}|{% err %}|File sizes greater than 4GB
+|{% err %}|{% err %}|{% err %}|Java version <= 6
+|{% ok %}|{% ok %}|{% err %}|Java version 7
+|{% ok %}|{% ok %}|{% ok %}|Java version 8
+|{% ok %}|{% ok %}|{% ok %}|Strict DFDL expressions adherence (i.e.
disallow features of XPath not available in DFDL expressions)
+|{% warn %}|{% warn %}|{% warn %}|XPath 2.0 Support (query-style expressions
not supported)
+
+| 1.0.0 | 1.1.0 | 2.0.0 |[DFDL v1.0 Spec
Errata](https://redmine.ogf.org/dmsf_files/13384?download=) (Minor Technical
Fixes)
+|:--------:|:--------:|:--------:|----------
+|{% err %}|{% err %}|{% err %}|Erratum 5.1
+|{% err %}|{% ok %}|{% ok %}|Erratum 5.2
+|{% err %}|{% err %}|{% err %}|Erratum 5.3
+|{% err %}|{% err %}|{% ok %}|Erratum 5.4
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.5
+|{% err %}|{% err %}|{% err %}|Erratum 5.6
+|{% err %}|{% err %}|{% ok %}|Erratum 5.7
+|{% err %}|{% err %}|{% ok %}|Erratum 5.8
+|{% err %}|{% err %}|{% ok %}|Erratum 5.9
+|{% err %}|{% err %}|{% ok %}|Erratum 5.10
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.11
+|{% err %}|{% err %}|{% ok %}|Erratum 5.12
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.13
+|{% err %}|{% err %}|{% err %}|Erratum 5.14
+|{% err %}|{% err %}|{% err %}|Erratum 5.15
+|{% err %}|{% err %}|{% err %}|Erratum 5.16
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.17
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.18
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.19
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.20
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.21
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.22
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.23
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.24
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.25
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.26
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.27
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.28
+|{% err %}|{% err %}|{% err %}|Erratum 5.29
+|{% err %}|{% err %}|{% err %}|Erratum 5.30
+|{% ok %}|{% ok %}|{% ok %}|Erratum 5.31
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
