This is an automated email from the ASF dual-hosted git repository.
slawrence pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/daffodil-sbt.git
The following commit(s) were added to refs/heads/main by this push:
new d9123b4 Require Scala 2.13.16 when for Daffodil 3.11.0
d9123b4 is described below
commit d9123b4804600ba164e8b5bce8b7d5108a59e25d
Author: Steve Lawrence <[email protected]>
AuthorDate: Fri Feb 20 11:25:11 2026 -0500
Require Scala 2.13.16 when for Daffodil 3.11.0
Scala 2.13.17 breaks serialization compatablity. Daffodil 3.11.0
requires scala 2.13.16, but the plugin current sets scalaVersion to
2.13.18. The causes packageDaffodilBin to create a saved parser that
cannot be used with Daffodil 3.11.0.
This issue seems to only affect Daffodil 3.11.0--all other versions of
Scala/Daffodil seem to maintain saved parser compatability.
This modifies the scalaVersion to pin the scala version to 2.13.16 when
Daffodil version is 3.11.0. This also modifies the sbt-daffodil-utils
project to not add a dependency to scala-library, instead assuming comes
from Daffodil or scalaVersion, allowing control over which libary
version is used.
Update the versions-01 test to include more complexity that is known to
cause issues with reloading saved parsers if built with newer versions
of scala-library.
Closes #163
---
build.sbt | 11 +++-
.../scala/org/apache/daffodil/DaffodilPlugin.scala | 42 +++++++++++----
.../src/main/resources/com/example/test.dfdl.xsd | 9 +++-
.../src/test/resources/com/example/input.txt | 1 +
.../src/test/resources/com/example/test.tdml | 2 +-
src/sbt-test/sbt-daffodil/versions-01/test.script | 60 ++++++++++++++++++----
6 files changed, 100 insertions(+), 25 deletions(-)
diff --git a/build.sbt b/build.sbt
index 4093db5..e1b3298 100644
--- a/build.sbt
+++ b/build.sbt
@@ -74,7 +74,8 @@ lazy val plugin = (project in file("."))
// Rat check settings
ratExcludes := Seq(
file(".git"),
- file("VERSION")
+ file("VERSION"),
+
file("src/sbt-test/sbt-daffodil/versions-01/src/test/resources/com/example/input.txt")
),
ratFailBinaries := true
)
@@ -120,7 +121,13 @@ lazy val utils = (projectMatrix in file("utils"))
)
}
}
- }
+ },
+ // Do not automatically add scala-library as a dependency to the
sbt-daffodil-utils jar. We
+ // expect scala-library to come from the provided Daffodil dependency.
This is important
+ // because in some cases newer versions of scala-library break
serialization compatability
+ // and the auto added version might be too new. When using
sbt-daffodil-utils, the
+ // sbt-daffodil plugin is careful about ensuring compatability
+ autoScalaLibrary := false
)
.jvmPlatform(
// We really only need Scala versions and not full ModuleIds, but using
ModuleId.revision
diff --git a/src/main/scala/org/apache/daffodil/DaffodilPlugin.scala
b/src/main/scala/org/apache/daffodil/DaffodilPlugin.scala
index 1fbb960..8a83f6e 100644
--- a/src/main/scala/org/apache/daffodil/DaffodilPlugin.scala
+++ b/src/main/scala/org/apache/daffodil/DaffodilPlugin.scala
@@ -205,22 +205,38 @@ object DaffodilPlugin extends AutoPlugin {
override lazy val projectSettings: Seq[Setting[_]] = Seq(
/**
- * Different versions of Daffodil depend on a specific major/minor version
of Scala. For
- * example, all versions of Daffodil 3.10.0 and older depend on Scala
2.12.x. Note that
- * Scala ensures binary compatibility, so we can use the latest patch
version for any
- * version of Daffodil, as long as we use the correct major/minor version.
These Scala
- * versions should be updated anytime a new patch release is made
available to ensure we
- * have the latest bug fixes and JDK compatibility. Note that this does
mean that we could
- * use a Scala version that a version of Daffodil was not released or
tested with, but Scala
- * backwards compatibility guarantees should prevent this from causing
issues.
+ * Each version of Daffodil is built and tested with a specific
major.minor.patch version of
+ * Scala. For example, Daffodil 3.10.0 was released using Scala 2.12.20.
Although Scala
+ * ensures binary compatibility with new patch releases, it does not
guarantee serialization
+ * compatibility, which is needed for saved parsers (for more information,
see
+ * https://github.com/scala/docs.scala-lang/issues/1202). However, we
sometimes need to use
+ * newer patch versions when serializing parsers to support newer versions
of Java. For
+ * example Scala 2.12.21 is required to build saved parsers on Java 25.
Although
+ * serialization compatibility is not guaranteed, in practice, with
Daffodils usage of Scala
+ * this seems to be safe and we are able to use the latest patch version
without issue.
+ *
+ * However, the one case where this is known to break is with Daffodil
3.11.0, which uses
+ * Scala 2.13.16, and where Scala 2.13.17 breaks serialization
compatibility. So for
+ * Daffodil 3.11.0, we pin the scalaVersion to 2.13.16. Note that this
could mean future
+ * version of Java will not be able to create saved parsers for 3.11.0.
+ *
+ * But for all other versions of Daffodil, we can use the latest patch
version and allow
+ * scala-steward to update them. This ensures we get the latest bug fixes
and JDK
+ * compatibility. If a new patch release breaks serialization
compatibility, we can pin it
+ * to the working version, which may limit JDK support.
*/
scalaVersion := {
// We really only need Scala versions and not full ModuleIds, but using
ModuleId.revision
// makes it easier for scala-steward to detect and automatically update
the versions when
// new Scala patch versions are released.
+ //
+ // Note that if we update one of the below scala-library versions, we
should verify that
+ // official Daffodil releases using older versions of scala-library can
still reload saved
+ // parsers built with this newer scalaVersion (see above note about
serialization
+ // compatibility). See the versions-01 scripted test to how to manually
verify this.
val daffodilToScalaVersion = Map(
">=4.0.0 " -> ("org.scala-lang" % "scala3-library" % "3.3.7").revision,
- "=3.11.0 " -> ("org.scala-lang" % "scala-library" %
"2.13.18").revision,
+ "=3.11.0 " -> ("org.scala-lang" % "scala-library" %
"2.13.16").revision, // scala-steward:off
"<=3.10.0" -> ("org.scala-lang" % "scala-library" % "2.12.21").revision
)
filterVersions(daffodilVersion.value, daffodilToScalaVersion).head
@@ -411,8 +427,12 @@ object DaffodilPlugin extends AutoPlugin {
libraryDependencies ++= {
daffodilPackageBinVersions.value.flatMap { binDaffodilVersion =>
val cfg = daffodilVersionId(binDaffodilVersion)
- // the Daffodil dependency must ignore the scalaVersion setting and
instead use
- // the specific version of scala used for the binDaffodilVersion.
+ // The daffodil-core/japi dependency transitively pulls in a
dependency to
+ // scala-library, while taking into account scalaVersion. Above, we
are careful to set
+ // scalaVersion appropriately to ensure compatibility with
scala-library. Also note that
+ // sbt-daffodil-utils is configured with autoScalaLibrary set to
false, so there are no
+ // concerns of it transitively pulling in a newer version of
scala-library that might
+ // break compatibility.
val daffodilToPackageBinDep = Map(
">=4.0.0 " -> Seq(
"org.apache.daffodil" % "daffodil-core_3" % binDaffodilVersion %
cfg,
diff --git
a/src/sbt-test/sbt-daffodil/versions-01/src/main/resources/com/example/test.dfdl.xsd
b/src/sbt-test/sbt-daffodil/versions-01/src/main/resources/com/example/test.dfdl.xsd
index dbab88c..444b899 100644
---
a/src/sbt-test/sbt-daffodil/versions-01/src/main/resources/com/example/test.dfdl.xsd
+++
b/src/sbt-test/sbt-daffodil/versions-01/src/main/resources/com/example/test.dfdl.xsd
@@ -21,6 +21,7 @@
xmlns:xs="http://www.w3.org/2001/XMLSchema";
xmlns:dfdl="http://www.ogf.org/dfdl/dfdl-1.0/";
xmlns:ex="http://example.com";
+ xmlns:fn="http://www.w3.org/2005/xpath-functions";
targetNamespace="http://example.com";
elementFormDefault="unqualified">
@@ -32,6 +33,12 @@
</appinfo>
</annotation>
- <element name="test01" type="xs:string" dfdl:lengthKind="delimited" />
+ <element name="test01" type="xs:string" dfdl:lengthKind="delimited"
dfdl:terminator="%NL;">
+ <annotation>
+ <appinfo source="http://www.ogf.org/dfdl/";>
+ <dfdl:assert test="{ fn:string-length(.) gt 0 }" />
+ </appinfo>
+ </annotation>
+ </element>
</schema>
diff --git
a/src/sbt-test/sbt-daffodil/versions-01/src/test/resources/com/example/input.txt
b/src/sbt-test/sbt-daffodil/versions-01/src/test/resources/com/example/input.txt
new file mode 100644
index 0000000..038d718
--- /dev/null
+++
b/src/sbt-test/sbt-daffodil/versions-01/src/test/resources/com/example/input.txt
@@ -0,0 +1 @@
+testing
diff --git
a/src/sbt-test/sbt-daffodil/versions-01/src/test/resources/com/example/test.tdml
b/src/sbt-test/sbt-daffodil/versions-01/src/test/resources/com/example/test.tdml
index c8d3e90..b2331a1 100644
---
a/src/sbt-test/sbt-daffodil/versions-01/src/test/resources/com/example/test.tdml
+++
b/src/sbt-test/sbt-daffodil/versions-01/src/test/resources/com/example/test.tdml
@@ -24,7 +24,7 @@
<parserTestCase name="test01" root="test01"
model="/com/example/test.dfdl.xsd">
<document>
- <documentPart type="text">testing</documentPart>
+ <documentPart type="file">input.txt</documentPart>
</document>
<infoset>
<dfdlInfoset>
diff --git a/src/sbt-test/sbt-daffodil/versions-01/test.script
b/src/sbt-test/sbt-daffodil/versions-01/test.script
index 3ebee0b..686e56f 100644
--- a/src/sbt-test/sbt-daffodil/versions-01/test.script
+++ b/src/sbt-test/sbt-daffodil/versions-01/test.script
@@ -16,74 +16,114 @@
## under the License.
##
+# Note that saved parsers are created and tested with the scalaVersion set by
+# the plugin. However, the scalaVersion can sometimes be different from the
+# actual scala-library that a version of Daffodil was built and tested with.
+# This is done intentionally to provide support for newer versions of Java, and
+# in most cases does not cause issues. But it is possible it could break
+# reloading saved parsers. So, if the plugin changes how scalaVersion is set,
+# we should uncomment the exec lines below to verify that the saved parsers
+# work with actual Daffodil releases that use possibly older scala-library
+# versions. Note that these lines are commented out because they add
+# significant time to test and really only need to be run if the scalaVersion
+# setting is updated. Note that these tests passing is not a guarantee that
+# newer Scala versions do not cause issues, but the schema used is designed to
+# detect known serialization issues.
+
> set daffodilVersion := "4.1.0"
> packageDaffodilBin
$ exists target/test-0.1-daffodil410.bin
> test
+# $ exec curl -L
https://www.apache.org/dyn/closer.lua/download/daffodil/4.1.0/bin/apache-daffodil-4.1.0-bin.tgz
-O --output-dir target/
+# $ exec tar -xvf target/apache-daffodil-4.1.0-bin.tgz -C target/
+# $ exec target/apache-daffodil-4.1.0-bin/bin/daffodil parse -P
target/test-0.1-daffodil410.bin src/test/resources/com/example/input.txt
> clean
> set daffodilVersion := "4.0.0"
> packageDaffodilBin
-$ exists target/test-0.1-daffodil400.bin
> test
+# $ exec curl -L
https://www.apache.org/dyn/closer.lua/download/daffodil/4.0.0/bin/apache-daffodil-4.0.0-bin.tgz
-O --output-dir target/
+# $ exec tar -xvf target/apache-daffodil-4.0.0-bin.tgz -C target/
+# $ exec target/apache-daffodil-4.0.0-bin/bin/daffodil parse -P
target/test-0.1-daffodil400.bin src/test/resources/com/example/input.txt
> clean
> set daffodilVersion := "3.11.0"
> packageDaffodilBin
$ exists target/test-0.1-daffodil3110.bin
> test
+# $ exec curl -L
https://www.apache.org/dyn/closer.lua/download/daffodil/3.11.0/bin/apache-daffodil-3.11.0-bin.tgz
-O --output-dir target/
+# $ exec tar -xvf target/apache-daffodil-3.11.0-bin.tgz -C target/
+# $ exec target/apache-daffodil-3.11.0-bin/bin/daffodil parse -P
target/test-0.1-daffodil3110.bin src/test/resources/com/example/input.txt
> clean
> set daffodilVersion := "3.10.0"
> packageDaffodilBin
-$ exists target/test-0.1-daffodil3100.bin
> test
+# $ exec curl -L
https://www.apache.org/dyn/closer.lua/download/daffodil/3.10.0/bin/apache-daffodil-3.10.0-bin.tgz
-O --output-dir target/
+# $ exec tar -xvf target/apache-daffodil-3.10.0-bin.tgz -C target/
+# $ exec target/apache-daffodil-3.10.0-bin/bin/daffodil parse -P
target/test-0.1-daffodil3100.bin src/test/resources/com/example/input.txt
> clean
> set daffodilVersion := "3.9.0"
> packageDaffodilBin
-$ exists target/test-0.1-daffodil390.bin
> test
+# $ exec curl -L
https://www.apache.org/dyn/closer.lua/download/daffodil/3.9.0/bin/apache-daffodil-3.9.0-bin.tgz
-O --output-dir target/
+# $ exec tar -xvf target/apache-daffodil-3.9.0-bin.tgz -C target/
+# $ exec target/apache-daffodil-3.9.0-bin/bin/daffodil parse -P
target/test-0.1-daffodil390.bin src/test/resources/com/example/input.txt
> clean
> set daffodilVersion := "3.8.0"
> packageDaffodilBin
-$ exists target/test-0.1-daffodil380.bin
> test
+# $ exec curl -L
https://www.apache.org/dyn/closer.lua/download/daffodil/3.8.0/bin/apache-daffodil-3.8.0-bin.tgz
-O --output-dir target/
+# $ exec tar -xvf target/apache-daffodil-3.8.0-bin.tgz -C target/
+# $ exec target/apache-daffodil-3.8.0-bin/bin/daffodil parse -P
target/test-0.1-daffodil380.bin src/test/resources/com/example/input.txt
> clean
> set daffodilVersion := "3.7.0"
> packageDaffodilBin
-$ exists target/test-0.1-daffodil370.bin
> test
+# $ exec curl -L
https://www.apache.org/dyn/closer.lua/download/daffodil/3.7.0/bin/apache-daffodil-3.7.0-bin.tgz
-O --output-dir target/
+# $ exec tar -xvf target/apache-daffodil-3.7.0-bin.tgz -C target/
+# $ exec target/apache-daffodil-3.7.0-bin/bin/daffodil parse -P
target/test-0.1-daffodil370.bin src/test/resources/com/example/input.txt
> clean
> set daffodilVersion := "3.6.0"
> packageDaffodilBin
-$ exists target/test-0.1-daffodil360.bin
> test
+# $ exec curl -L
https://www.apache.org/dyn/closer.lua/download/daffodil/3.6.0/bin/apache-daffodil-3.6.0-bin.tgz
-O --output-dir target/
+# $ exec tar -xvf target/apache-daffodil-3.6.0-bin.tgz -C target/
+# $ exec target/apache-daffodil-3.6.0-bin/bin/daffodil parse -P
target/test-0.1-daffodil360.bin src/test/resources/com/example/input.txt
> clean
> set daffodilVersion := "3.5.0"
> packageDaffodilBin
-$ exists target/test-0.1-daffodil350.bin
> test
+# $ exec curl -L
https://www.apache.org/dyn/closer.lua/download/daffodil/3.5.0/bin/apache-daffodil-3.5.0-bin.tgz
-O --output-dir target/
+# $ exec tar -xvf target/apache-daffodil-3.5.0-bin.tgz -C target/
+# $ exec target/apache-daffodil-3.5.0-bin/bin/daffodil parse -P
target/test-0.1-daffodil350.bin src/test/resources/com/example/input.txt
> clean
> set daffodilVersion := "3.4.0"
> packageDaffodilBin
-$ exists target/test-0.1-daffodil340.bin
> test
+# $ exec curl -L
https://www.apache.org/dyn/closer.lua/download/daffodil/3.4.0/bin/apache-daffodil-3.4.0-bin.tgz
-O --output-dir target/
+# $ exec tar -xvf target/apache-daffodil-3.4.0-bin.tgz -C target/
+# $ exec target/apache-daffodil-3.4.0-bin/bin/daffodil parse -P
target/test-0.1-daffodil340.bin src/test/resources/com/example/input.txt
> clean
> set daffodilVersion := "3.3.0"
> packageDaffodilBin
-$ exists target/test-0.1-daffodil330.bin
> test
+# $ exec curl -L
https://www.apache.org/dyn/closer.lua/download/daffodil/3.3.0/bin/apache-daffodil-3.3.0-bin.tgz
-O --output-dir target/
+# $ exec tar -xvf target/apache-daffodil-3.3.0-bin.tgz -C target/
+# $ exec target/apache-daffodil-3.3.0-bin/bin/daffodil parse -P
target/test-0.1-daffodil330.bin src/test/resources/com/example/input.txt
> clean
> set daffodilVersion := "3.2.0"
> packageDaffodilBin
-$ exists target/test-0.1-daffodil320.bin
> test
+# $ exec curl -L
https://www.apache.org/dyn/closer.lua/download/daffodil/3.2.0/bin/apache-daffodil-3.2.0-bin.tgz
-O --output-dir target/
+# $ exec tar -xvf target/apache-daffodil-3.2.0-bin.tgz -C target/
+# $ exec target/apache-daffodil-3.2.0-bin/bin/daffodil parse -P
target/test-0.1-daffodil320.bin src/test/resources/com/example/input.txt
> clean
