stevedlawrence commented on code in PR #183:
URL: https://github.com/apache/daffodil-sbt/pull/183#discussion_r3222464679
##########
src/main/scala/org/apache/daffodil/DaffodilPlugin.scala:
##########
@@ -840,6 +858,198 @@ object DaffodilPlugin extends AutoPlugin {
}
}
+ def daffodilFlattenSettings: Seq[Setting[_]] = Seq(
+ daffodilFlattenTarget := target.value /
s"${name.value}-${version.value}-flat.zip",
+ /* Paths in daffodilFlattenExcludes/Includes that are not globbed at the
start are
+ * generally only going to match paths within JAR files on the classpath.
+ * In order to deal with paths on the filesystem we need to glob the start
+ * of the path to account for different directory structure before the
+ * schema project path.
+ */
+ daffodilFlattenIncludes := "*.xsd" | "*.xsl" | "*.xslt" | "*.xml",
+ daffodilFlattenExcludes := HiddenFileFilter,
+
+ /**
+ * Whether or not to publish the flattened schemas zip. Defaults to false.
+ *
+ * If projects want to publish flattened schemas then they must explicitly
enable it by
+ * setting 'daffodilFlattenSchemas / publishArtifact := true'.
+ *
+ * If false, flattened schemas will not be created unless you explicitly
run the
+ * daffodilFlattenSchemas.
+ */
+ daffodilFlattenSchemas / publishArtifact := false,
+
+ daffodilFlattenSchemas / artifact := Artifact(
+ name.value,
+ "flat",
+ "zip",
+ Some("flat"),
+ Vector(),
+ None
+ ),
+ daffodilFlattenSchemas := {
+
+ val logger = streams.value.log
+ val filter = daffodilFlattenIncludes.value --
daffodilFlattenExcludes.value
+
+ val flatDir = target.value / "flatDir"
+ if (flatDir.exists())
+ IO.delete(flatDir)
+ IO.createDirectory(flatDir)
+
+ val projectResources = (Compile / resourceDirectories).value
+ .map { root => root.toURI.toURL -> (root **
filter).get.map(_.toURI.toURL).toList }
+ .toMap
+
+ /* Get all dependency jars and resources specific to this project. Note
+ * that the "Test" configuration is used for JAR files in order to ensure
+ * we pull in XSD files from daffodil-lib, as in many schema projects the
+ * daffodil dependencies are only used for testing, not compiling. Also
+ * note that we want to use Test/externalDependencyClasspath to get
+ * dependency jars, and not something like Test/fullClasspath or
+ * Test/dependencyClasspath, since those could trigger expensive resource
+ * generators or compilation of internal test jars that we don't need--we
+ * only need Compile/resources from this project and Test/dependency jars
+ * from external projects
+ */
+ val projectURLs = (Compile /
resourceDirectories).value.map(_.toURI.toURL)
+ val allClasspathURLs = (Compile /
fullClasspath).value.map(_.data.toURI.toURL)
+ val classLoader = new URLClassLoader((projectURLs ++
allClasspathURLs).toArray, null)
+
+ val schemaLocationPattern = """schemaLocation=\"([^\"]*)\"""".r
+ val hrefPattern = "href=\"([^\"]*)\"".r
+ val documentPattern = "document[(]'([^']*)'[)]".r
+ val referenceRegexes = List(schemaLocationPattern, hrefPattern,
documentPattern)
+
+ val referencedFiles = {
+ // This annotation simply warns if the compiler cannot enable tail
call optimization
+ @scala.annotation.tailrec
+ def getReferences(root: URL, parents: Seq[URL], acc:
ArrayBuffer[URL]): Seq[URL] = {
+ val rootPath = Paths.get(root.toURI)
+ parents match {
+ // Already processed this file (h), proceed with rest of files (t)
+ case h :: t if (acc.contains(h)) =>
+ getReferences(
+ root,
+ t,
+ acc
+ )
+ // Need to process this file (h)
+ case h :: t => {
+ val relPath = h.toString match {
+ case j if (j.startsWith("jar:")) =>
Paths.get(j.toString.split("!")(1).tail)
+ case f if (f.startsWith("file:")) =>
rootPath.relativize(Paths.get(h.toURI))
+ }
+ val bytes = h.openStream().readAllBytes()
+ val newPath = Paths.get(
+ flatDir.toString,
+ relPath.toString.replaceAll("/", "__")
+ )
+ val bw = Files.newBufferedWriter(newPath)
+ val fileAsString = new String(bytes, Charset.defaultCharset())
+ val references = {
+ referenceRegexes.flatMap { re =>
+ re.findAllIn(fileAsString).matchData.map(_.group(1))
+ }
+ }
+
+ // Resolve the location of all references on the actual file
+ // system
+ val resolvedReferences = references
+ .map { ref =>
+ val origLocation = {
+ if (ref.contains("urn:")) {
+ val urn = ref.split(" ")(1)
+ if (urn.startsWith("/"))
+ urn.tail
+ else
+ urn
+ } else if (ref.startsWith("/"))
+ ref.tail
+ else
+ ref
+ }
+ val res = {
+ val r = classLoader.findResource(origLocation)
+ if (r == null) {
+ val rrel = List(relPath.getParent,
origLocation).mkString("/")
+ classLoader.findResource(rrel)
Review Comment:
As an example of my first paragraph, say we have resolved some include
statement to the jar url:
```
jar:///some/schema/format.jar!/com/example/foo.dfdl.xsd
```
And say that foo.dfdl.xsd has a relative include like this:
```xml
<include schemaLocation="bar.dfdl.xsd" />
```
In this case, because the schemaLocation is a relative path Daffodil
resolves bar.dfdl.xsd to a file inside format.jar, and it should resolve to
this:
```
jar:///some/schema/format.jar!/com/example/bar.dfdl.xsd
```
What this logic does is it first resolves bar.dfdl.xsd path relative to the
root `/com/example/foo.dfdl.xsd` to get `/com/example/bar.dfdl.xsd` and then
looks for that on the classpath. Although this works most of the time, in
theory it is possible for the resource `/com/example/bar/dfdl.xsd` to exist in
multiple jars, and instead of finding the one in format.jar we might find one
in some other jar if it appears earlier on the classpath, and so it would
resolve to a file in a different jar.
So if a path is relative, instead of building a resource path and looking
for it on the classpath, we should instead resolve the path relative to the
importing file. We really should only ever resolve via the class loader if the
schemaLocation starts with a slash and so is absolute.
Note that Daffodil does have deprecated behavior where it if it doesn't find
a relative schemaLocation using the above logic then it tries to resolve the
relative path as it it were absolute (i.e. using the classpath), so we might
want that as fall back to match that behavior. So I think the logic wants to be
something like:
```scala
if (origLocation.startsWith("/")) {
// absolute path, find it using the class loader with the slash removed
classLoader.findResource(origLocation.tail)
} else {
// relative path
// first try to find it relative to the importing file without looking at
the classpath
// may need different logic depending on if importingFile is a jar:// or a
file://
val res = origLocation.resolvePathRelativeTo(importingFile)
if (res != null) {
res
} else {
// if that failed, try the deprecated behavior of resolving relative
paths as if they were absolute
classLoader.findResource(origLocation)
}
}
```
Here is where daffodil implements this logic, though it's a bit more complex
since it also supports edge cases that I don't think this plugin needs to
support:
https://github.com/apache/daffodil/blob/main/daffodil-core/src/main/scala/org/apache/daffodil/lib/xml/XMLUtils.scala#L1518-L1579
----
Somewhat related, this made me think of another potential issue: In theory
two separate jars could contain the same path and we could resolve to both of
them separately from different files, but they would flatten to the same thing.
For example, you could have something like
```
jar:///header.jar!/header.dfdl.xsd
jar:///header.jar!/common.dfdl.xsd
jar:///payload.jar!/payload.dfdl.xsd
jar:///payload.jar!/common.dfdl.xsd
```
Where header.dfdl.xsd references the common.dfdl.xsd in header.jar, and
payload.dfdl.xsd references common.dfdl.xsd in payload.jar. Although this
should be avoided, it wouldn't surprise me if we actually have some schemas
that do something like this, especilaly since we have lotsof schemas that are
glue schemas that combine things like headers and payloads together. And in
this case we would flatten to a single common.dfdl.xsd file even if the files
were different and one would probably just overwrite the other?
I wonder if the solution for the flattener is if a file is in a jar then we
also include the jar name? So for example, the files above files flatten to:
```
header__header.dfdl.xsd
header__common.dfdl.xsd
payload__payload.dfdl.xsd
payload__common.dfdl.xsd
```
This still could break if jar files with the same name on the classpath, but
that's probably alot less likely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
