stevedlawrence commented on code in PR #183:
URL: https://github.com/apache/daffodil-sbt/pull/183#discussion_r3219124927
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -63,6 +68,18 @@ object DaffodilPlugin extends AutoPlugin {
val daffodilTdmlUsesPackageBin = settingKey[Boolean](
"Whether or not TDML files use the saved parsers created by
daffodilPackageBin"
)
+ val daffodilFlattenTarget = settingKey[File](
+ "File to write the flattened schemas package to"
+ )
+ val daffodilFlattenSchemas = taskKey[File](
+ "flatten the directory structure of all schemas and schema dependencies
to a single common directory and update 'schemaLocation' paths to match"
+ )
+ val daffodilFlattenIncludes = settingKey[FileFilter](
+ "File extensions to include when flattening resources. Defaults to *.xsd
| *.xsl | *.xslt | *.xml"
+ )
+ val daffodilFlattenExcludes = settingKey[FileFilter](
+ "Globs of paths to exclude from schema flattening"
+ )
Review Comment:
I think we can remove these custom settings. The idiomatic sbt way to use
includeFilter/excludeFilter is to scope it to the task that uses it. For
example, SBT defines includeFilter for a number of different tasks:
```scala
unmanagedSources / includeFilter :== ("*.java" | "*.scala"),
unmanagedJars / includeFilter :== "*.jar" | "*.so" | "*.dll" |
"*.jnilib" | "*.zip",
unmanagedResources / includeFilter :== AllPassFilter,
```
So we can remove these vals and instead change the settings below can contain
```scala
daffodilFlattenSchemas / includeFilter := "*.xsd" | "*.xsl" | "*.xslt" |
"*.xml",
daffodilFlattenSchemas / excludeFilter := HiddenFileFilter,
```
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
Review Comment:
Is this comment correct now? It mentions the start of paths but I don't
think these default globs do anything related to start of paths now
It might be worth mentioning the the filters are run to match resource
files, and then just say something like those files (and anything they
reference, which could be found in dependency jars) are flattened.
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
+
+ val logger = streams.value.log
+ val filter = daffodilFlattenIncludes.value --
daffodilFlattenExcludes.value
+
+ val flatDir = target.value / "flatDir"
+ if (flatDir.exists())
+ IO.delete(flatDir)
+ IO.createDirectory(flatDir)
+
+ val projectResources = (Compile / resourceDirectories).value
+ .map { root => root.toURI.toURL -> (root **
filter).get.map(_.toURI.toURL).toList }
+ .toMap
+
+ /* Get all dependency jars and resources specific to this project. Note
+ * that the "Test" configuration is used for JAR files in order to ensure
+ * we pull in XSD files from daffodil-lib, as in many schema projects the
+ * daffodil dependencies are only used for testing, not compiling. Also
+ * note that we want to use Test/externalDependencyClasspath to get
+ * dependency jars, and not something like Test/fullClasspath or
+ * Test/dependencyClasspath, since those could trigger expensive resource
+ * generators or compilation of internal test jars that we don't need--we
+ * only need Compile/resources from this project and Test/dependency jars
+ * from external projects
+ */
+ val projectURLs = (Compile /
resourceDirectories).value.map(_.toURI.toURL)
+ val allClasspathURLs = (Compile /
fullClasspath).value.map(_.data.toURI.toURL)
+ val classLoader = new URLClassLoader((projectURLs ++
allClasspathURLs).toArray, null)
+
+ val schemaLocationPattern = """schemaLocation=\"([^\"]*)\"""".r
+ val hrefPattern = "href=\"([^\"]*)\"".r
+ val documentPattern = "document[(]'([^']*)'[)]".r
+ val referenceRegexes = List(schemaLocationPattern, hrefPattern,
documentPattern)
+
+ val referencedFiles = {
+ // This annotation simply warns if the compiler cannot enable tail
call optimization
+ @scala.annotation.tailrec
+ def getReferences(root: URL, parents: Seq[URL], acc:
ArrayBuffer[URL]): Seq[URL] = {
+ val rootPath = Paths.get(root.toURI)
+ parents match {
+ // Already processed this file (h), proceed with rest of files (t)
+ case h :: t if (acc.contains(h)) =>
+ getReferences(
+ root,
+ t,
+ acc
+ )
+ // Need to process this file (h)
+ case h :: t => {
+ val relPath = h.toString match {
+ case j if (j.startsWith("jar:")) =>
Paths.get(j.toString.split("!")(1).tail)
+ case f if (f.startsWith("file:")) =>
rootPath.relativize(Paths.get(h.toURI))
+ }
+ val bytes = h.openStream().readAllBytes()
+ val newPath = Paths.get(
+ flatDir.toString,
+ relPath.toString.replaceAll("/", "__")
+ )
+ val bw = Files.newBufferedWriter(newPath)
+ val fileAsString = new String(bytes, Charset.defaultCharset())
+ val references = {
+ referenceRegexes.flatMap { re =>
+ re.findAllIn(fileAsString).matchData.map(_.group(1))
+ }
+ }
+
+ // Resolve the location of all references on the actual file
+ // system
+ val resolvedReferences = references
+ .map { ref =>
+ val origLocation = {
+ if (ref.contains("urn:")) {
Review Comment:
Is this to match schemaLocation attributes that are of the form
`xsi:schemaLocation="<namespace> <path>"`?
I don't think the namespace needs to start with "urn:", it could be any
valid namespace. Note also that xsi:schemaLocation supports multiple locations,
e.g.
```xml
<schema
xsi:schemaLocation="http://example.com/ exampleOrg.xsd
http://example.org/ exampleCom.xsd">
```
So maybe the right thing is if it contains a space then we split it and take
every other path?
Are there other places that use syntax that could contain a space?
Alliteratively, I don't think we really use `xsi:schemaLocation`, so we
could just say this isn't supported and modify `schemaLocationPattern` so it
does not match anything that has a prefix, since the include/import statements
do not have namespace prefix for their schemaLocation attribute.
##########
VERSION:
##########
@@ -1 +1 @@
-1.7.0
+1.8.0-SNAPSHOT
Review Comment:
Please revert this change. We have a separate open PR for this.
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
+
+ val logger = streams.value.log
+ val filter = daffodilFlattenIncludes.value --
daffodilFlattenExcludes.value
+
+ val flatDir = target.value / "flatDir"
+ if (flatDir.exists())
+ IO.delete(flatDir)
+ IO.createDirectory(flatDir)
+
+ val projectResources = (Compile / resourceDirectories).value
+ .map { root => root.toURI.toURL -> (root **
filter).get.map(_.toURI.toURL).toList }
+ .toMap
+
+ /* Get all dependency jars and resources specific to this project. Note
+ * that the "Test" configuration is used for JAR files in order to ensure
+ * we pull in XSD files from daffodil-lib, as in many schema projects the
+ * daffodil dependencies are only used for testing, not compiling. Also
+ * note that we want to use Test/externalDependencyClasspath to get
+ * dependency jars, and not something like Test/fullClasspath or
+ * Test/dependencyClasspath, since those could trigger expensive resource
+ * generators or compilation of internal test jars that we don't need--we
+ * only need Compile/resources from this project and Test/dependency jars
+ * from external projects
+ */
+ val projectURLs = (Compile /
resourceDirectories).value.map(_.toURI.toURL)
+ val allClasspathURLs = (Compile /
fullClasspath).value.map(_.data.toURI.toURL)
+ val classLoader = new URLClassLoader((projectURLs ++
allClasspathURLs).toArray, null)
+
+ val schemaLocationPattern = """schemaLocation=\"([^\"]*)\"""".r
+ val hrefPattern = "href=\"([^\"]*)\"".r
+ val documentPattern = "document[(]'([^']*)'[)]".r
+ val referenceRegexes = List(schemaLocationPattern, hrefPattern,
documentPattern)
+
+ val referencedFiles = {
+ // This annotation simply warns if the compiler cannot enable tail
call optimization
+ @scala.annotation.tailrec
+ def getReferences(root: URL, parents: Seq[URL], acc:
ArrayBuffer[URL]): Seq[URL] = {
+ val rootPath = Paths.get(root.toURI)
+ parents match {
+ // Already processed this file (h), proceed with rest of files (t)
+ case h :: t if (acc.contains(h)) =>
+ getReferences(
+ root,
+ t,
+ acc
+ )
+ // Need to process this file (h)
+ case h :: t => {
+ val relPath = h.toString match {
+ case j if (j.startsWith("jar:")) =>
Paths.get(j.toString.split("!")(1).tail)
+ case f if (f.startsWith("file:")) =>
rootPath.relativize(Paths.get(h.toURI))
+ }
+ val bytes = h.openStream().readAllBytes()
+ val newPath = Paths.get(
+ flatDir.toString,
+ relPath.toString.replaceAll("/", "__")
+ )
+ val bw = Files.newBufferedWriter(newPath)
+ val fileAsString = new String(bytes, Charset.defaultCharset())
+ val references = {
+ referenceRegexes.flatMap { re =>
+ re.findAllIn(fileAsString).matchData.map(_.group(1))
+ }
+ }
+
+ // Resolve the location of all references on the actual file
+ // system
+ val resolvedReferences = references
+ .map { ref =>
+ val origLocation = {
+ if (ref.contains("urn:")) {
+ val urn = ref.split(" ")(1)
+ if (urn.startsWith("/"))
+ urn.tail
+ else
+ urn
+ } else if (ref.startsWith("/"))
+ ref.tail
+ else
+ ref
+ }
+ val res = {
+ val r = classLoader.findResource(origLocation)
+ if (r == null) {
+ val rrel = List(relPath.getParent,
origLocation).mkString("/")
+ classLoader.findResource(rrel)
+ } else
+ r
+ }
+ if (res == null)
+ logger.warn(s"Unable to resolve reference to $ref from
source file $h")
+ ref -> Option(res)
+ }
+ .toMap
+ .filter(e => e._2.isDefined)
+
+ // For each reference do a search and replace of the entire file
+ val updatedFileAsString = {
+ resolvedReferences.foldLeft(fileAsString) {
+ case (input, (ref, resolvedRef)) => {
+ // For each reference replace each instance of it in the
+ // file with the same reference but with "/" changed to
"__"
+ val rref = resolvedRef.get.toString
+ val relativized = rref match {
+ case j if (j.startsWith("jar")) => j.split("!")(1).tail
+ case f if (f.startsWith("file")) =>
rootPath.relativize(Paths.get(resolvedRef.get.toURI)).toString
+ }
+ input.replaceAll(ref, relativized.replaceAll("/", "__"))
+ }
+ }
+ }
+
+ bw.write(updatedFileAsString)
+ bw.close()
+
+ /* Have succesfully processed this file (h), call getReferences
+ * again on the rest of the list (t) + any references from this
+ * file. Add this file to the accumulator list
+ */
+ getReferences(
+ root,
+ t ++ resolvedReferences.values
+ .collect { case Some(url) => url }
+ .filterNot(t.contains),
Review Comment:
We might consider making the second parameter of getReferences a set instead
of Seq. Set's automatically handle efficiently ignoring duplicates. This could
just be sometihng like `t ++ resolvedReferences.values`, so it's simpler and
more efficient.
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
+
+ val logger = streams.value.log
+ val filter = daffodilFlattenIncludes.value --
daffodilFlattenExcludes.value
+
+ val flatDir = target.value / "flatDir"
+ if (flatDir.exists())
+ IO.delete(flatDir)
+ IO.createDirectory(flatDir)
+
+ val projectResources = (Compile / resourceDirectories).value
+ .map { root => root.toURI.toURL -> (root **
filter).get.map(_.toURI.toURL).toList }
+ .toMap
+
+ /* Get all dependency jars and resources specific to this project. Note
+ * that the "Test" configuration is used for JAR files in order to ensure
+ * we pull in XSD files from daffodil-lib, as in many schema projects the
+ * daffodil dependencies are only used for testing, not compiling. Also
+ * note that we want to use Test/externalDependencyClasspath to get
+ * dependency jars, and not something like Test/fullClasspath or
+ * Test/dependencyClasspath, since those could trigger expensive resource
+ * generators or compilation of internal test jars that we don't need--we
+ * only need Compile/resources from this project and Test/dependency jars
+ * from external projects
+ */
+ val projectURLs = (Compile /
resourceDirectories).value.map(_.toURI.toURL)
+ val allClasspathURLs = (Compile /
fullClasspath).value.map(_.data.toURI.toURL)
+ val classLoader = new URLClassLoader((projectURLs ++
allClasspathURLs).toArray, null)
+
+ val schemaLocationPattern = """schemaLocation=\"([^\"]*)\"""".r
+ val hrefPattern = "href=\"([^\"]*)\"".r
+ val documentPattern = "document[(]'([^']*)'[)]".r
+ val referenceRegexes = List(schemaLocationPattern, hrefPattern,
documentPattern)
+
+ val referencedFiles = {
+ // This annotation simply warns if the compiler cannot enable tail
call optimization
+ @scala.annotation.tailrec
+ def getReferences(root: URL, parents: Seq[URL], acc:
ArrayBuffer[URL]): Seq[URL] = {
+ val rootPath = Paths.get(root.toURI)
+ parents match {
+ // Already processed this file (h), proceed with rest of files (t)
+ case h :: t if (acc.contains(h)) =>
+ getReferences(
+ root,
+ t,
+ acc
+ )
+ // Need to process this file (h)
+ case h :: t => {
+ val relPath = h.toString match {
+ case j if (j.startsWith("jar:")) =>
Paths.get(j.toString.split("!")(1).tail)
+ case f if (f.startsWith("file:")) =>
rootPath.relativize(Paths.get(h.toURI))
+ }
+ val bytes = h.openStream().readAllBytes()
+ val newPath = Paths.get(
+ flatDir.toString,
+ relPath.toString.replaceAll("/", "__")
+ )
+ val bw = Files.newBufferedWriter(newPath)
+ val fileAsString = new String(bytes, Charset.defaultCharset())
+ val references = {
+ referenceRegexes.flatMap { re =>
+ re.findAllIn(fileAsString).matchData.map(_.group(1))
+ }
+ }
+
+ // Resolve the location of all references on the actual file
+ // system
+ val resolvedReferences = references
+ .map { ref =>
+ val origLocation = {
+ if (ref.contains("urn:")) {
+ val urn = ref.split(" ")(1)
+ if (urn.startsWith("/"))
+ urn.tail
+ else
+ urn
+ } else if (ref.startsWith("/"))
+ ref.tail
+ else
+ ref
+ }
+ val res = {
+ val r = classLoader.findResource(origLocation)
+ if (r == null) {
+ val rrel = List(relPath.getParent,
origLocation).mkString("/")
+ classLoader.findResource(rrel)
+ } else
+ r
+ }
+ if (res == null)
+ logger.warn(s"Unable to resolve reference to $ref from
source file $h")
+ ref -> Option(res)
+ }
+ .toMap
+ .filter(e => e._2.isDefined)
+
+ // For each reference do a search and replace of the entire file
+ val updatedFileAsString = {
+ resolvedReferences.foldLeft(fileAsString) {
+ case (input, (ref, resolvedRef)) => {
+ // For each reference replace each instance of it in the
+ // file with the same reference but with "/" changed to
"__"
+ val rref = resolvedRef.get.toString
+ val relativized = rref match {
+ case j if (j.startsWith("jar")) => j.split("!")(1).tail
+ case f if (f.startsWith("file")) =>
rootPath.relativize(Paths.get(resolvedRef.get.toURI)).toString
+ }
+ input.replaceAll(ref, relativized.replaceAll("/", "__"))
+ }
+ }
+ }
+
+ bw.write(updatedFileAsString)
+ bw.close()
+
+ /* Have succesfully processed this file (h), call getReferences
+ * again on the rest of the list (t) + any references from this
+ * file. Add this file to the accumulator list
+ */
+ getReferences(
+ root,
+ t ++ resolvedReferences.values
+ .collect { case Some(url) => url }
+ .filterNot(t.contains),
+ acc += h
+ )
+ }
+ case _ => acc // Have processed all referenced files, return acc
+ }
+ }
+ val acc = new ArrayBuffer[URL]()
+ projectResources foreach { case (root, files) =>
+ getReferences(root, files, acc)
+ }
Review Comment:
getReferences seems to be wrrtten with a goal of avoiding mutable data
strctures, excdpt for this ArrayBuffer. If we want to avoid the immutable
ArrayBuffer and make this more Scala-y, I think you could change getReferences
to something like this and use foldLeft (and changing to Set's as mentioned in
other comments):
def getReferences(op: ((URL, Set[URL]), Set[URL])): Set[URL] = {
val ((root, parents), accumulator) = op
...
accumulator
}
val referencedFiles =
projectResources.foldLeft(Set.empty[URL])(getReferences)
As long as each branch of getReferences returns the accumulator I think this
works, but all the recursion and accumulators does make this a bit hard to
follow.
Maybe a potential alternative, is to make more use of mutable data
structures and a while loop. It could avoids recursion and the need for
tail-loop optimization to be efficient, at the cost of not being very
functional. For example something like this:
```scala
// a place to store files that need to be processed for references
val unprocessedFiles = mutable.Queue[URL](projectResources: _*)
// a place to store each file that have already seen seen. It's either files
that we have already procesed or files that are already queue to be proceseed.
Initialized with the files in the queue to be processed.
val seenFiles = mutable.Set[URL](projectResources: _*)
while (unprocessedFiles.nonEmpty) {
val file = unprocessedFiles.dequeue()
val references = ... // get all references from the file
// for any found references, add them to the seen set. If we haven't
already seen
// them before, add them to the queue for later processing
references.foreach { ref =>
if (!seen.add(ref)) unprocessesFiles.queue(ref)
}
// write file with its references updated
...
// done with this file, loop around and process any unprocessed files
}
// at this point unprocessedFiles is empty and seenFiles contains all the
URLs that were processed
```
Despite the non-functional nature of this, I personally find it easier to
follow and verify its correctness. Making sure the accumulator is updated
correctly and the logic is tail-recursive with the other approach is a bit more
difficult.
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
Review Comment:
Something we learned about with `packageDaffodilBin` is it's helpful to use
the `products` task to do the actual work, and then the main task references
that. So this might want to be something like:
```scala
daffodilFlattenSchemas / products := {
// all this stuff
},
daffodilFlattenSchemas := {
(daffodilFlattenSchemas / products).value.head
}
```
The benefit is that internally things can access `(daffodilFlattenSchemas /
products).value`, with the impliciation that `daffodilFlattenSchemas` is only
ever run manually. In some cases this allows us to give different diagnostics
depending on how things are triggered. For example packageDaffodilBin can give
warnings if settings aren't correct. I'm not sure we need that for this task,
but it makes it an option if we ever need it.
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
+
+ val logger = streams.value.log
+ val filter = daffodilFlattenIncludes.value --
daffodilFlattenExcludes.value
+
Review Comment:
We might want to consider adding support for caching in this task. I imagine
with large schemas the flatten work to scan all files could take a noticable
amount of time? And SBT makes it relatively easy.
The pattern would be something like:
```scala
val cachedDir = streams.value.cacheDirectory / "daffodilFlattenSchemas"
val cachedFun = FileFunction.cached(cachedDir) { (_: Set[File]) =>
// existingLogic
...
daffodilFlattenTarget.value
}
cachedFun(projectResources ++ allClasspathURLs)
```
So this creates a cache function that does all the work, and then tells the
cache function it needs to be rerun if any of the input files (i.e. project
resource or classpath) changes.
We do something very similar for packageDaffodilBin
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
+
+ val logger = streams.value.log
+ val filter = daffodilFlattenIncludes.value --
daffodilFlattenExcludes.value
+
+ val flatDir = target.value / "flatDir"
+ if (flatDir.exists())
+ IO.delete(flatDir)
+ IO.createDirectory(flatDir)
+
+ val projectResources = (Compile / resourceDirectories).value
+ .map { root => root.toURI.toURL -> (root **
filter).get.map(_.toURI.toURL).toList }
+ .toMap
+
+ /* Get all dependency jars and resources specific to this project. Note
+ * that the "Test" configuration is used for JAR files in order to ensure
+ * we pull in XSD files from daffodil-lib, as in many schema projects the
+ * daffodil dependencies are only used for testing, not compiling. Also
+ * note that we want to use Test/externalDependencyClasspath to get
+ * dependency jars, and not something like Test/fullClasspath or
+ * Test/dependencyClasspath, since those could trigger expensive resource
+ * generators or compilation of internal test jars that we don't need--we
+ * only need Compile/resources from this project and Test/dependency jars
+ * from external projects
Review Comment:
Does this comment need updating? It mentions using the `Test` configuration
and other settings that it no longer uses.
That said, are those still needed? I don't think `Compile/fullClasspath`
will contain things like the daffodil-lib XSDs. Maybe
Test/externalDependencyClasspath is the correct thing to use instead of
Compile/fullClasspath?
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
+
+ val logger = streams.value.log
+ val filter = daffodilFlattenIncludes.value --
daffodilFlattenExcludes.value
+
+ val flatDir = target.value / "flatDir"
+ if (flatDir.exists())
+ IO.delete(flatDir)
+ IO.createDirectory(flatDir)
+
+ val projectResources = (Compile / resourceDirectories).value
+ .map { root => root.toURI.toURL -> (root **
filter).get.map(_.toURI.toURL).toList }
+ .toMap
+
+ /* Get all dependency jars and resources specific to this project. Note
+ * that the "Test" configuration is used for JAR files in order to ensure
+ * we pull in XSD files from daffodil-lib, as in many schema projects the
+ * daffodil dependencies are only used for testing, not compiling. Also
+ * note that we want to use Test/externalDependencyClasspath to get
+ * dependency jars, and not something like Test/fullClasspath or
+ * Test/dependencyClasspath, since those could trigger expensive resource
+ * generators or compilation of internal test jars that we don't need--we
+ * only need Compile/resources from this project and Test/dependency jars
+ * from external projects
+ */
+ val projectURLs = (Compile /
resourceDirectories).value.map(_.toURI.toURL)
+ val allClasspathURLs = (Compile /
fullClasspath).value.map(_.data.toURI.toURL)
+ val classLoader = new URLClassLoader((projectURLs ++
allClasspathURLs).toArray, null)
+
+ val schemaLocationPattern = """schemaLocation=\"([^\"]*)\"""".r
+ val hrefPattern = "href=\"([^\"]*)\"".r
+ val documentPattern = "document[(]'([^']*)'[)]".r
+ val referenceRegexes = List(schemaLocationPattern, hrefPattern,
documentPattern)
+
+ val referencedFiles = {
+ // This annotation simply warns if the compiler cannot enable tail
call optimization
+ @scala.annotation.tailrec
+ def getReferences(root: URL, parents: Seq[URL], acc:
ArrayBuffer[URL]): Seq[URL] = {
+ val rootPath = Paths.get(root.toURI)
+ parents match {
+ // Already processed this file (h), proceed with rest of files (t)
+ case h :: t if (acc.contains(h)) =>
+ getReferences(
+ root,
+ t,
+ acc
+ )
+ // Need to process this file (h)
+ case h :: t => {
+ val relPath = h.toString match {
+ case j if (j.startsWith("jar:")) =>
Paths.get(j.toString.split("!")(1).tail)
+ case f if (f.startsWith("file:")) =>
rootPath.relativize(Paths.get(h.toURI))
+ }
+ val bytes = h.openStream().readAllBytes()
+ val newPath = Paths.get(
+ flatDir.toString,
+ relPath.toString.replaceAll("/", "__")
+ )
+ val bw = Files.newBufferedWriter(newPath)
+ val fileAsString = new String(bytes, Charset.defaultCharset())
+ val references = {
+ referenceRegexes.flatMap { re =>
+ re.findAllIn(fileAsString).matchData.map(_.group(1))
+ }
+ }
+
+ // Resolve the location of all references on the actual file
+ // system
+ val resolvedReferences = references
+ .map { ref =>
+ val origLocation = {
+ if (ref.contains("urn:")) {
+ val urn = ref.split(" ")(1)
+ if (urn.startsWith("/"))
+ urn.tail
+ else
+ urn
+ } else if (ref.startsWith("/"))
+ ref.tail
+ else
+ ref
+ }
+ val res = {
+ val r = classLoader.findResource(origLocation)
+ if (r == null) {
+ val rrel = List(relPath.getParent,
origLocation).mkString("/")
+ classLoader.findResource(rrel)
Review Comment:
In Daffodil, relative paths are scanned relative to the file that
include/imported them and do not reach out to the classpath. So I think instead
of using the classLoader here we just want to look at the including path mutate
it. So if the including file was in a jar then we need to ensure the relative
path is appended to the original jar path. And similarly if it's a file.
Also, sometimes the including path has a relative location like
`../../foo.dfdl.xsd`, so we might want to normalize the path to remove those
kinds of things since i"m not sure if they're valid in URLs.
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
+
+ val logger = streams.value.log
+ val filter = daffodilFlattenIncludes.value --
daffodilFlattenExcludes.value
+
+ val flatDir = target.value / "flatDir"
+ if (flatDir.exists())
+ IO.delete(flatDir)
+ IO.createDirectory(flatDir)
+
+ val projectResources = (Compile / resourceDirectories).value
+ .map { root => root.toURI.toURL -> (root **
filter).get.map(_.toURI.toURL).toList }
+ .toMap
+
+ /* Get all dependency jars and resources specific to this project. Note
+ * that the "Test" configuration is used for JAR files in order to ensure
+ * we pull in XSD files from daffodil-lib, as in many schema projects the
+ * daffodil dependencies are only used for testing, not compiling. Also
+ * note that we want to use Test/externalDependencyClasspath to get
+ * dependency jars, and not something like Test/fullClasspath or
+ * Test/dependencyClasspath, since those could trigger expensive resource
+ * generators or compilation of internal test jars that we don't need--we
+ * only need Compile/resources from this project and Test/dependency jars
+ * from external projects
+ */
+ val projectURLs = (Compile /
resourceDirectories).value.map(_.toURI.toURL)
+ val allClasspathURLs = (Compile /
fullClasspath).value.map(_.data.toURI.toURL)
+ val classLoader = new URLClassLoader((projectURLs ++
allClasspathURLs).toArray, null)
+
+ val schemaLocationPattern = """schemaLocation=\"([^\"]*)\"""".r
+ val hrefPattern = "href=\"([^\"]*)\"".r
+ val documentPattern = "document[(]'([^']*)'[)]".r
+ val referenceRegexes = List(schemaLocationPattern, hrefPattern,
documentPattern)
+
+ val referencedFiles = {
+ // This annotation simply warns if the compiler cannot enable tail
call optimization
+ @scala.annotation.tailrec
+ def getReferences(root: URL, parents: Seq[URL], acc:
ArrayBuffer[URL]): Seq[URL] = {
+ val rootPath = Paths.get(root.toURI)
+ parents match {
+ // Already processed this file (h), proceed with rest of files (t)
+ case h :: t if (acc.contains(h)) =>
+ getReferences(
+ root,
+ t,
+ acc
+ )
+ // Need to process this file (h)
+ case h :: t => {
+ val relPath = h.toString match {
+ case j if (j.startsWith("jar:")) =>
Paths.get(j.toString.split("!")(1).tail)
+ case f if (f.startsWith("file:")) =>
rootPath.relativize(Paths.get(h.toURI))
+ }
+ val bytes = h.openStream().readAllBytes()
+ val newPath = Paths.get(
+ flatDir.toString,
+ relPath.toString.replaceAll("/", "__")
+ )
+ val bw = Files.newBufferedWriter(newPath)
+ val fileAsString = new String(bytes, Charset.defaultCharset())
+ val references = {
+ referenceRegexes.flatMap { re =>
+ re.findAllIn(fileAsString).matchData.map(_.group(1))
+ }
+ }
+
+ // Resolve the location of all references on the actual file
+ // system
+ val resolvedReferences = references
+ .map { ref =>
+ val origLocation = {
+ if (ref.contains("urn:")) {
+ val urn = ref.split(" ")(1)
+ if (urn.startsWith("/"))
+ urn.tail
+ else
+ urn
+ } else if (ref.startsWith("/"))
+ ref.tail
+ else
+ ref
+ }
+ val res = {
+ val r = classLoader.findResource(origLocation)
+ if (r == null) {
+ val rrel = List(relPath.getParent,
origLocation).mkString("/")
+ classLoader.findResource(rrel)
+ } else
+ r
+ }
+ if (res == null)
+ logger.warn(s"Unable to resolve reference to $ref from
source file $h")
+ ref -> Option(res)
Review Comment:
This is minor, but it might be a bit simpler to change this map to a
.flatMap and return an `Option[(String, URL)]` instead of `(String,
Option[URL])`. The flatMap will remove the None's and you can remove the
.filter and other places where you have to deal with the None case.
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
+
+ val logger = streams.value.log
+ val filter = daffodilFlattenIncludes.value --
daffodilFlattenExcludes.value
+
+ val flatDir = target.value / "flatDir"
+ if (flatDir.exists())
+ IO.delete(flatDir)
+ IO.createDirectory(flatDir)
+
+ val projectResources = (Compile / resourceDirectories).value
+ .map { root => root.toURI.toURL -> (root **
filter).get.map(_.toURI.toURL).toList }
+ .toMap
+
+ /* Get all dependency jars and resources specific to this project. Note
+ * that the "Test" configuration is used for JAR files in order to ensure
+ * we pull in XSD files from daffodil-lib, as in many schema projects the
+ * daffodil dependencies are only used for testing, not compiling. Also
+ * note that we want to use Test/externalDependencyClasspath to get
+ * dependency jars, and not something like Test/fullClasspath or
+ * Test/dependencyClasspath, since those could trigger expensive resource
+ * generators or compilation of internal test jars that we don't need--we
+ * only need Compile/resources from this project and Test/dependency jars
+ * from external projects
+ */
+ val projectURLs = (Compile /
resourceDirectories).value.map(_.toURI.toURL)
+ val allClasspathURLs = (Compile /
fullClasspath).value.map(_.data.toURI.toURL)
+ val classLoader = new URLClassLoader((projectURLs ++
allClasspathURLs).toArray, null)
+
+ val schemaLocationPattern = """schemaLocation=\"([^\"]*)\"""".r
+ val hrefPattern = "href=\"([^\"]*)\"".r
+ val documentPattern = "document[(]'([^']*)'[)]".r
+ val referenceRegexes = List(schemaLocationPattern, hrefPattern,
documentPattern)
+
+ val referencedFiles = {
+ // This annotation simply warns if the compiler cannot enable tail
call optimization
+ @scala.annotation.tailrec
+ def getReferences(root: URL, parents: Seq[URL], acc:
ArrayBuffer[URL]): Seq[URL] = {
Review Comment:
Might want to consider making the accumulator a Set? We only need fast
append and and fast lookup which a Set gives us.
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
+
+ val logger = streams.value.log
+ val filter = daffodilFlattenIncludes.value --
daffodilFlattenExcludes.value
+
+ val flatDir = target.value / "flatDir"
+ if (flatDir.exists())
+ IO.delete(flatDir)
+ IO.createDirectory(flatDir)
+
+ val projectResources = (Compile / resourceDirectories).value
+ .map { root => root.toURI.toURL -> (root **
filter).get.map(_.toURI.toURL).toList }
+ .toMap
+
+ /* Get all dependency jars and resources specific to this project. Note
+ * that the "Test" configuration is used for JAR files in order to ensure
+ * we pull in XSD files from daffodil-lib, as in many schema projects the
+ * daffodil dependencies are only used for testing, not compiling. Also
+ * note that we want to use Test/externalDependencyClasspath to get
+ * dependency jars, and not something like Test/fullClasspath or
+ * Test/dependencyClasspath, since those could trigger expensive resource
+ * generators or compilation of internal test jars that we don't need--we
+ * only need Compile/resources from this project and Test/dependency jars
+ * from external projects
+ */
+ val projectURLs = (Compile /
resourceDirectories).value.map(_.toURI.toURL)
+ val allClasspathURLs = (Compile /
fullClasspath).value.map(_.data.toURI.toURL)
+ val classLoader = new URLClassLoader((projectURLs ++
allClasspathURLs).toArray, null)
+
+ val schemaLocationPattern = """schemaLocation=\"([^\"]*)\"""".r
+ val hrefPattern = "href=\"([^\"]*)\"".r
+ val documentPattern = "document[(]'([^']*)'[)]".r
+ val referenceRegexes = List(schemaLocationPattern, hrefPattern,
documentPattern)
+
+ val referencedFiles = {
+ // This annotation simply warns if the compiler cannot enable tail
call optimization
Review Comment:
Does this lead to an error since we treat warnings as errors? I'm concerned
that this function is quite complex so it wouldn't surprise me if we
accidentally make changes that break tail recursion and cause issues with
deeply nested and self-referential files
Alternatively, I imagine this could be pretty cleanly rewritten without tail
recursion.
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
+
+ val logger = streams.value.log
+ val filter = daffodilFlattenIncludes.value --
daffodilFlattenExcludes.value
+
+ val flatDir = target.value / "flatDir"
+ if (flatDir.exists())
+ IO.delete(flatDir)
+ IO.createDirectory(flatDir)
+
+ val projectResources = (Compile / resourceDirectories).value
+ .map { root => root.toURI.toURL -> (root **
filter).get.map(_.toURI.toURL).toList }
+ .toMap
+
+ /* Get all dependency jars and resources specific to this project. Note
+ * that the "Test" configuration is used for JAR files in order to ensure
+ * we pull in XSD files from daffodil-lib, as in many schema projects the
+ * daffodil dependencies are only used for testing, not compiling. Also
+ * note that we want to use Test/externalDependencyClasspath to get
+ * dependency jars, and not something like Test/fullClasspath or
+ * Test/dependencyClasspath, since those could trigger expensive resource
+ * generators or compilation of internal test jars that we don't need--we
+ * only need Compile/resources from this project and Test/dependency jars
+ * from external projects
+ */
+ val projectURLs = (Compile /
resourceDirectories).value.map(_.toURI.toURL)
+ val allClasspathURLs = (Compile /
fullClasspath).value.map(_.data.toURI.toURL)
+ val classLoader = new URLClassLoader((projectURLs ++
allClasspathURLs).toArray, null)
+
+ val schemaLocationPattern = """schemaLocation=\"([^\"]*)\"""".r
+ val hrefPattern = "href=\"([^\"]*)\"".r
+ val documentPattern = "document[(]'([^']*)'[)]".r
+ val referenceRegexes = List(schemaLocationPattern, hrefPattern,
documentPattern)
+
+ val referencedFiles = {
+ // This annotation simply warns if the compiler cannot enable tail
call optimization
+ @scala.annotation.tailrec
+ def getReferences(root: URL, parents: Seq[URL], acc:
ArrayBuffer[URL]): Seq[URL] = {
Review Comment:
Is there a name we could use other than `parents`? Usually when I see
parents I expect so also see children, so it's a bit confusioning. Or maybe
renaming someting else to make it clear what the parent/child relationship is?
Or mmaybe we call these 'unprocessedFiles' and the accumulator is renamed
'processedFiles' to make it more clear what these things represent?
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
+
+ val logger = streams.value.log
+ val filter = daffodilFlattenIncludes.value --
daffodilFlattenExcludes.value
+
+ val flatDir = target.value / "flatDir"
+ if (flatDir.exists())
+ IO.delete(flatDir)
+ IO.createDirectory(flatDir)
+
+ val projectResources = (Compile / resourceDirectories).value
+ .map { root => root.toURI.toURL -> (root **
filter).get.map(_.toURI.toURL).toList }
+ .toMap
+
+ /* Get all dependency jars and resources specific to this project. Note
+ * that the "Test" configuration is used for JAR files in order to ensure
+ * we pull in XSD files from daffodil-lib, as in many schema projects the
+ * daffodil dependencies are only used for testing, not compiling. Also
+ * note that we want to use Test/externalDependencyClasspath to get
+ * dependency jars, and not something like Test/fullClasspath or
+ * Test/dependencyClasspath, since those could trigger expensive resource
+ * generators or compilation of internal test jars that we don't need--we
+ * only need Compile/resources from this project and Test/dependency jars
+ * from external projects
+ */
+ val projectURLs = (Compile /
resourceDirectories).value.map(_.toURI.toURL)
+ val allClasspathURLs = (Compile /
fullClasspath).value.map(_.data.toURI.toURL)
+ val classLoader = new URLClassLoader((projectURLs ++
allClasspathURLs).toArray, null)
+
+ val schemaLocationPattern = """schemaLocation=\"([^\"]*)\"""".r
+ val hrefPattern = "href=\"([^\"]*)\"".r
+ val documentPattern = "document[(]'([^']*)'[)]".r
+ val referenceRegexes = List(schemaLocationPattern, hrefPattern,
documentPattern)
+
+ val referencedFiles = {
+ // This annotation simply warns if the compiler cannot enable tail
call optimization
+ @scala.annotation.tailrec
+ def getReferences(root: URL, parents: Seq[URL], acc:
ArrayBuffer[URL]): Seq[URL] = {
+ val rootPath = Paths.get(root.toURI)
+ parents match {
+ // Already processed this file (h), proceed with rest of files (t)
+ case h :: t if (acc.contains(h)) =>
+ getReferences(
+ root,
+ t,
+ acc
+ )
+ // Need to process this file (h)
+ case h :: t => {
+ val relPath = h.toString match {
+ case j if (j.startsWith("jar:")) =>
Paths.get(j.toString.split("!")(1).tail)
+ case f if (f.startsWith("file:")) =>
rootPath.relativize(Paths.get(h.toURI))
+ }
+ val bytes = h.openStream().readAllBytes()
+ val newPath = Paths.get(
+ flatDir.toString,
+ relPath.toString.replaceAll("/", "__")
+ )
+ val bw = Files.newBufferedWriter(newPath)
+ val fileAsString = new String(bytes, Charset.defaultCharset())
+ val references = {
+ referenceRegexes.flatMap { re =>
+ re.findAllIn(fileAsString).matchData.map(_.group(1))
+ }
+ }
+
+ // Resolve the location of all references on the actual file
+ // system
+ val resolvedReferences = references
+ .map { ref =>
+ val origLocation = {
+ if (ref.contains("urn:")) {
+ val urn = ref.split(" ")(1)
+ if (urn.startsWith("/"))
+ urn.tail
+ else
+ urn
+ } else if (ref.startsWith("/"))
+ ref.tail
+ else
+ ref
+ }
+ val res = {
+ val r = classLoader.findResource(origLocation)
+ if (r == null) {
+ val rrel = List(relPath.getParent,
origLocation).mkString("/")
+ classLoader.findResource(rrel)
+ } else
+ r
+ }
+ if (res == null)
+ logger.warn(s"Unable to resolve reference to $ref from
source file $h")
+ ref -> Option(res)
+ }
+ .toMap
+ .filter(e => e._2.isDefined)
+
+ // For each reference do a search and replace of the entire file
+ val updatedFileAsString = {
+ resolvedReferences.foldLeft(fileAsString) {
+ case (input, (ref, resolvedRef)) => {
+ // For each reference replace each instance of it in the
+ // file with the same reference but with "/" changed to
"__"
+ val rref = resolvedRef.get.toString
+ val relativized = rref match {
+ case j if (j.startsWith("jar")) => j.split("!")(1).tail
+ case f if (f.startsWith("file")) =>
rootPath.relativize(Paths.get(resolvedRef.get.toURI)).toString
+ }
+ input.replaceAll(ref, relativized.replaceAll("/", "__"))
+ }
+ }
+ }
+
+ bw.write(updatedFileAsString)
+ bw.close()
+
+ /* Have succesfully processed this file (h), call getReferences
+ * again on the rest of the list (t) + any references from this
+ * file. Add this file to the accumulator list
+ */
+ getReferences(
+ root,
Review Comment:
Thinking about this root stuff some more, I'm not sure the way we handle
root is correct. For example, say we have two files in two separate resource
directories in a project. The most common example would be managed and
unmanaged resources, e.g.
```
unmanagedResourceDir/foo.dfdl.xsd
managedResourceDir/bar.dfdl.xsd
```
As we resolve references in foo.dfdl.xsd, root will be set to
unmanagedResourceDir. And say foo.dfdl.xsd references bar.dfdl.xsd. I think in
this case we will relativize bar.dfdl.xsd relative to foo's root, which is
different and things will break.
One option is if something resolves to file then do not recursively call
getReferences, with the assumption that it should exist in projectReferences
and it's root should be correcte. Though, I'm not positive that assumption is
always true, especially if users do weird things with excludeFilter, which
might exclude a file.
Another option is if something resolves to a file then we just check if the
files starts with all possible resource directories and remove which ever it
starts with. It's a bit inefficient, but it should work. The only case I can
think it would break is if a resource directory is a subdirectory of another
resource directory and so a file could potentially star with multiple resource
directories, but I don't think that should ever happen--feels like a project
doing that would have lots of other issues if it had nested resource dirs.
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +859,288 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def flattenerSettings: Seq[Setting[_]] = Seq(
+ flattenTarget := target.value / s"${name.value}-${version.value}-flat.zip",
+ /* Paths in flattenExcludes/Includes that are not globbed at the start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ flattenExcludes := Seq(
+ // "**/src/test/resources/**",
+ // "Log4j*.xsd",
+ // "xsd/**", // This is coming from XSAT2
+ // "IBMdefined/**",
+ // "org/apache/xml/**",
+ // "edu/illinois/ncsa/daffodil/**",
+ "org/apache/daffodil/**"
+ // "**/*-tests.jar",
+ // "META-INF/**",
+ // "com/ibm/icu/**",
+ // "eclipse-xml-catalog.xml",
+ // "daffodil-built-in-catalog.xml"
+ ),
+ flattenIncludes := Seq(
+ "org/apache/daffodil/xsd/DFDLGeneralFormat*.dfdl.xsd"
+ ),
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'flattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * flattenSchemas.
+ */
+ flattenSchemas / publishArtifact := false,
+
+ flattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ flattenSchemas := {
+
+ val logger = streams.value.log
+
+ val extractDir = Paths.get(target.value.getPath(), "flatExtractDir")
+ if (Files.exists(extractDir))
+ IO.delete(extractDir.toFile)
+ Files.createDirectory(extractDir)
+
+ val flatDir = Paths.get(target.value.getPath(), "flatDir")
+ if (Files.exists(flatDir))
+ IO.delete(flatDir.toFile)
+ Files.createDirectory(flatDir)
+
+ val projectXsdFiles = (Compile / resourceDirectories).value
+ .flatMap { dir => (dir ** "*.xsd").get }
+ .map(path => Paths.get(path.toString))
+ val projectXslFiles = (Compile / resourceDirectories).value
+ .flatMap { dir => (dir ** ("*.xsl" || "*.xslt")).get }
+ .map(path => Paths.get(path.toString))
+ val projectXmlFiles = (Compile / resourceDirectories).value
+ .flatMap { dir => (dir ** "*.xml").get }
+ .map(path => Paths.get(path.toString))
+
+ /* Copy schema files from the current project's src/main/resources
+ * directory
+ */
+ val filesFromProject =
+ (projectXsdFiles ++ projectXslFiles ++ projectXmlFiles).filterNot {
path =>
+ val matchesExcludes = flattenExcludes.value.exists(glob =>
glob.matches(path))
+ val matchesIncludes = flattenIncludes.value.exists(glob =>
glob.matches(path))
+ matchesExcludes && !matchesIncludes
+ }
+
+ val extractedProjectFiles = filesFromProject.map { origPath =>
+ val resourcePath = Paths
+ .get((Compile / resourceDirectories).value(0).toString)
+ .relativize(origPath)
+ .toString
+ val newPath = Paths.get(extractDir.toString, resourcePath)
+ Files.createDirectories(newPath.getParent())
+ Files.copy(origPath, newPath, StandardCopyOption.REPLACE_EXISTING)
+ newPath
+ }
+
+ /* Get all dependency jars and resources specific to this project. Note
+ * that the "Test" configuration is used for JAR files in order to ensure
+ * we pull in XSD files from daffodil-lib, as in many schema projects the
+ * daffodil dependencies are only used for testing, not compiling. Also
+ * note that we want to use Test/externalDependencyClasspath to get
+ * dependency jars, and not something like Test/fullClasspath or
+ * Test/dependencyClasspath, since those could trigger expensive resource
+ * generators or compilation of internal test jars that we don't need--we
+ * only need Compile/resources from this project and Test/dependency jars
+ * from external projects
+ */
+ val projectJarFiles = (Test /
externalDependencyClasspath).value.files.flatMap { file =>
+ (file ** "*.jar").get
+ }
+
+ projectJarFiles.reverse.map { jar =>
+ val env = new HashMap[String, String]
+ val fs = FileSystems.newFileSystem(URI.create(s"jar:file:${jar}"), env)
+ val rootDirs = fs.getRootDirectories().asScala
+ rootDirs.foreach { dir =>
+ val xsdFiles =
Files.walk(dir).iterator().asScala.filter(_.toString.endsWith(".xsd"))
+
+ // Includes XML files as they may be used for configuration of the
XSLT
+ val xsltFiles = Files
+ .walk(dir)
+ .iterator()
+ .asScala
+ .toList
+ .filter(f =>
+ f.toString.endsWith(".xslt") || f.toString.endsWith(".xsl") ||
f.toString
+ .endsWith(".xml")
+ )
+
+ /* For each XSD file we have, we want to extract it from its original
+ * jar while maintaining its path from inside the jar, ex:
+ *
com.owlcyberdefense.whatever.jar:com/owlcyberdefense/whatever/xsd/whatever.xsd
+ *
+ * extracts to
+ *
+ *
target/flatExtractDir/com/owlcyberdefense/whatever/xsd/whatever.xsd
+ *
+ * We also want to exclude schemas from paths listed in
flattenExcludes
+ * that are not also listed in flattenIncludes
+ */
+ val filesToExtract = (xsdFiles ++ xsltFiles).filterNot { path =>
+ val matchesExcludes = flattenExcludes.value.exists(glob =>
glob.matches(path))
+ val matchesIncludes = flattenIncludes.value.exists(glob =>
glob.matches(path))
+ matchesExcludes && !matchesIncludes
+ }
+
+ filesToExtract.foreach { origPath =>
+ val newPath = Paths.get(extractDir.toString, origPath.toString)
+ Files.createDirectories(newPath.getParent())
+ Files.copy(origPath, newPath, StandardCopyOption.REPLACE_EXISTING)
+ }
+ }
+ fs.close()
+ }
+
+ val schemaLocationPattern = """schemaLocation=\"([^\"]*)\"""".r
+ val hrefPattern = "href=\"([^\"]*)\"".r
+ val documentPattern = "document[(]'([^']*)'[)]".r
+ val referenceRegexes = List(schemaLocationPattern, hrefPattern,
documentPattern)
Review Comment:
Thoughts on making these a configurable setting? I doubt anyone would really
need to change these, but with includeFilter now configurable users could
include other files that might have different references.
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
+
+ val logger = streams.value.log
+ val filter = daffodilFlattenIncludes.value --
daffodilFlattenExcludes.value
+
+ val flatDir = target.value / "flatDir"
+ if (flatDir.exists())
+ IO.delete(flatDir)
+ IO.createDirectory(flatDir)
+
+ val projectResources = (Compile / resourceDirectories).value
+ .map { root => root.toURI.toURL -> (root **
filter).get.map(_.toURI.toURL).toList }
+ .toMap
+
+ /* Get all dependency jars and resources specific to this project. Note
+ * that the "Test" configuration is used for JAR files in order to ensure
+ * we pull in XSD files from daffodil-lib, as in many schema projects the
+ * daffodil dependencies are only used for testing, not compiling. Also
+ * note that we want to use Test/externalDependencyClasspath to get
+ * dependency jars, and not something like Test/fullClasspath or
+ * Test/dependencyClasspath, since those could trigger expensive resource
+ * generators or compilation of internal test jars that we don't need--we
+ * only need Compile/resources from this project and Test/dependency jars
+ * from external projects
+ */
+ val projectURLs = (Compile /
resourceDirectories).value.map(_.toURI.toURL)
+ val allClasspathURLs = (Compile /
fullClasspath).value.map(_.data.toURI.toURL)
+ val classLoader = new URLClassLoader((projectURLs ++
allClasspathURLs).toArray, null)
+
+ val schemaLocationPattern = """schemaLocation=\"([^\"]*)\"""".r
+ val hrefPattern = "href=\"([^\"]*)\"".r
+ val documentPattern = "document[(]'([^']*)'[)]".r
+ val referenceRegexes = List(schemaLocationPattern, hrefPattern,
documentPattern)
+
+ val referencedFiles = {
+ // This annotation simply warns if the compiler cannot enable tail
call optimization
+ @scala.annotation.tailrec
+ def getReferences(root: URL, parents: Seq[URL], acc:
ArrayBuffer[URL]): Seq[URL] = {
+ val rootPath = Paths.get(root.toURI)
+ parents match {
+ // Already processed this file (h), proceed with rest of files (t)
+ case h :: t if (acc.contains(h)) =>
+ getReferences(
+ root,
+ t,
+ acc
+ )
+ // Need to process this file (h)
+ case h :: t => {
+ val relPath = h.toString match {
+ case j if (j.startsWith("jar:")) =>
Paths.get(j.toString.split("!")(1).tail)
+ case f if (f.startsWith("file:")) =>
rootPath.relativize(Paths.get(h.toURI))
+ }
+ val bytes = h.openStream().readAllBytes()
+ val newPath = Paths.get(
+ flatDir.toString,
+ relPath.toString.replaceAll("/", "__")
+ )
+ val bw = Files.newBufferedWriter(newPath)
+ val fileAsString = new String(bytes, Charset.defaultCharset())
+ val references = {
+ referenceRegexes.flatMap { re =>
+ re.findAllIn(fileAsString).matchData.map(_.group(1))
+ }
+ }
+
+ // Resolve the location of all references on the actual file
+ // system
+ val resolvedReferences = references
+ .map { ref =>
+ val origLocation = {
+ if (ref.contains("urn:")) {
+ val urn = ref.split(" ")(1)
+ if (urn.startsWith("/"))
+ urn.tail
+ else
+ urn
+ } else if (ref.startsWith("/"))
+ ref.tail
+ else
+ ref
+ }
+ val res = {
+ val r = classLoader.findResource(origLocation)
+ if (r == null) {
+ val rrel = List(relPath.getParent,
origLocation).mkString("/")
+ classLoader.findResource(rrel)
+ } else
+ r
+ }
+ if (res == null)
+ logger.warn(s"Unable to resolve reference to $ref from
source file $h")
+ ref -> Option(res)
+ }
+ .toMap
+ .filter(e => e._2.isDefined)
+
+ // For each reference do a search and replace of the entire file
+ val updatedFileAsString = {
+ resolvedReferences.foldLeft(fileAsString) {
+ case (input, (ref, resolvedRef)) => {
+ // For each reference replace each instance of it in the
+ // file with the same reference but with "/" changed to
"__"
+ val rref = resolvedRef.get.toString
+ val relativized = rref match {
+ case j if (j.startsWith("jar")) => j.split("!")(1).tail
+ case f if (f.startsWith("file")) =>
rootPath.relativize(Paths.get(resolvedRef.get.toURI)).toString
+ }
+ input.replaceAll(ref, relativized.replaceAll("/", "__"))
+ }
+ }
+ }
+
+ bw.write(updatedFileAsString)
+ bw.close()
+
+ /* Have succesfully processed this file (h), call getReferences
+ * again on the rest of the list (t) + any references from this
+ * file. Add this file to the accumulator list
+ */
+ getReferences(
+ root,
+ t ++ resolvedReferences.values
+ .collect { case Some(url) => url }
+ .filterNot(t.contains),
+ acc += h
+ )
+ }
+ case _ => acc // Have processed all referenced files, return acc
+ }
+ }
+ val acc = new ArrayBuffer[URL]()
+ projectResources foreach { case (root, files) =>
+ getReferences(root, files, acc)
+ }
+ acc
+ }
+
+ /* Create zip file containing all flattened schemas */
+ val flattenedFiles = IO.listFiles(flatDir)
+ val sources = flattenedFiles.map(file => file -> file.getName())
+ IO.zip(sources, daffodilFlattenTarget.value,
Some(System.currentTimeMillis()))
Review Comment:
currentTimeMillis makes it difficult to create reproducible zips. That's not
a strict requirement, but it would be nice if it just worked. I'd suggest we
use `Package.defaultTimestamp`, which is provided by SBT and will use
SOURCE_DATE_EPOCH if it's defined or some constant timestamp if not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
