Thank you for sharing more details and suggestions, Vlad and Herman. Additionally, I found one interesting thread where an Apache Spark PMC member asked the specific question to `legal-discuss@` literally for the above `TestSerDe.jar` file.
The thread at https://lists.apache.org/list?legal-disc...@apache.org - https://lists.apache.org/thread/xmbgpgt30n7fdd99pnbg7983qzzrx24k (June 25, 2018) (Re: LICENSE and NOTICE file content) Apache Spark 1.0 (jar) - https://github.com/apache/spark/blob/branch-1.0/sql/hive/src/test/resources/data/files/TestSerDe.jar Apache Hive 1.2 (Original Source Code) - https://github.com/apache/hive/blob/rel/release-1.2.2/itests/test-serde/src/main/java/org/apache/hadoop/hive/serde2/TestSerDe.java Dongjoon. On 2025/02/26 04:59:56 "Rozov, Vlad" wrote: > AFAIK, there are 3 ways to satisfy ASF source release policy: > > 1. Use source code to build test jars and remove jars from the source release > and they may still be part of the source in the github repo (preferable). > 2. Publish jars as artifacts in Maven central and pull them as a dependency. > 3. Repackage jars to include LICENSE. > > Thank you, > > Vlad > > On Feb 25, 2025, at 7:18 PM, Herman van Hovell > <her...@databricks.com.INVALID> wrote: > > https://github.com/apache/spark/blob/master/sql/connect/client/jvm/src/test/resources/StubClassDummyUdf.scala > (commit 408c3df) is the source code for the following jars: > > * ./connector/connect/server/src/test/resources/udf_noA.jar > * ./connector/connect/client/jvm/src/test/resources/udf2.13.jar > * ./connector/connect/client/jvm/src/test/resources/udf2.12.jar > > https://github.com/apache/spark/blob/master/sql/hive/src/test/resources/regression-test-SPARK-8489/Main.scala > and > https://github.com/apache/spark/blob/master/sql/hive/src/test/resources/regression-test-SPARK-8489/MyCoolClass.scala > are the source files for the following jars: > > * ./sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.12.jar > * ./sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.13.jar > * ./sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar > > ./sql/core/src/test/resources/SPARK-33084.jar was added in > https://github.com/apache/spark/commit/771c538620e66be2d0fb0e383e4aa37b4d29f7eb. > The commit contains the source code. > > I will get the source code for the following jars checked in: > > * ./core/src/test/resources/TestHelloV3_2.12.jar > * ./core/src/test/resources/TestHelloV2_2.12.jar > * ./core/src/test/resources/TestHelloV2_2.13.jar > * ./core/src/test/resources/TestHelloV3_2.13.jar > * > ./connector/connect/common/src/test/resources/artifact-tests/junitLargeJar.jar > * > ./connector/connect/common/src/test/resources/artifact-tests/smallJar.jar > * ./connector/connect/client/jvm/src/test/resources/TestHelloV2_2.12.jar > * ./connector/connect/client/jvm/src/test/resources/TestHelloV2_2.13.jar > * ./data/artifact-tests/junitLargeJar.jar > * ./data/artifact-tests/smallJar.jar > > > On Tue, Feb 25, 2025 at 9:22 PM Rozov, Vlad <vro...@amazon.com.invalid> wrote: > Right, the issue does not seem to be new for 3.5 and it is not new for 3.5.5. > Here is the list of all jars I found in the source release: > > ./core/src/test/resources/TestHelloV3_2.12.jar > ./core/src/test/resources/TestHelloV2_2.12.jar > ./core/src/test/resources/TestHelloV2_2.13.jar > ./core/src/test/resources/TestHelloV3_2.13.jar > ./core/src/test/resources/TestUDTF.jar > ./connector/connect/server/src/test/resources/udf_noA.jar > ./connector/connect/common/src/test/resources/artifact-tests/junitLargeJar.jar > ./connector/connect/common/src/test/resources/artifact-tests/smallJar.jar > ./connector/connect/client/jvm/src/test/resources/TestHelloV2_2.12.jar > ./connector/connect/client/jvm/src/test/resources/TestHelloV2_2.13.jar > ./connector/connect/client/jvm/src/test/resources/udf2.13.jar > ./connector/connect/client/jvm/src/test/resources/udf2.12.jar > ./data/artifact-tests/junitLargeJar.jar > ./data/artifact-tests/smallJar.jar > ./sql/core/src/test/resources/SPARK-33084.jar > ./sql/hive/src/test/noclasspath/hive-test-udfs.jar > ./sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.12.jar > ./sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.13.jar > ./sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar > ./sql/hive/src/test/resources/SPARK-21101-1.0.jar > ./sql/hive/src/test/resources/data/files/TestSerDe.jar > ./sql/hive/src/test/resources/TestUDTF.jar > ./sql/hive-thriftserver/src/test/resources/TestUDTF.jar > > Thank you, > > Vlad > > > On Feb 25, 2025, at 3:03 PM, Dongjoon Hyun > > <dongj...@apache.org<mailto:dongj...@apache.org>> wrote: > > > > Just for the record, `TestUDTF.jar` test resource seems to be one of the > > oldest examples which Vlad pointed. > > > > https://github.com/apache/spark/blob/v2.0.0/core/src/test/resources/TestUDTF.jar > > (Added 9 years ago) > > > > $ tar tvf spark-2.0.0.tgz | grep TestUDTF.jar > > -rw-r--r-- 0 jenkins jenkins 1328 Jul 19 2016 > > spark-2.0.0/sql/hive/src/test/resources/TestUDTF.jar > > -rw-r--r-- 0 jenkins jenkins 1328 Jul 19 2016 > > spark-2.0.0/sql/hive-thriftserver/src/test/resources/TestUDTF.jar > > -rw-r--r-- 0 jenkins jenkins 1328 Jul 19 2016 > > spark-2.0.0/core/src/test/resources/TestUDTF.jar > > > > Dongjoon. > > > > > > On 2025/02/25 22:47:07 Dongjoon Hyun wrote: > >> Thank you for reply, Vlad. > >> > >> Do you mean SPARK-44246 violated ASF policy since Apache Spark 3.5.0 by > >> injecting a unlicensed test jar, `TestHelloV2.jar`? > >> > >> https://issues.apache.org/jira/browse/SPARK-44246 > >> https://github.com/apache/spark/pull/41789 > >> > >> And, it was spread to `TestHelloV3_2.13.jar` via SPARK-44297 in the same > >> way? > >> > >> https://issues.apache.org/jira/browse/SPARK-44297 > >> https://github.com/apache/spark/pull/41852 > >> > >> > >> Dongjoon. > >> > >> On 2025/02/25 19:53:45 "Rozov, Vlad" wrote: > >>> I am not sure if this was already discussed and noted, so want to confirm > >>> with PMC members: > >>> > >>> I see several (test) JAR files included into the ASF source release that > >>> do not have LICENSE in the MANIFEST or META-INF and do not have the > >>> source code. For example core/src/test/resources/TestHelloV3_2.13.jar. > >>> AFAIK it may be against ASF source release policy to include such jars > >>> into the source release (see "Does the policy apply to binary/object > >>> files, such as executables or JAR files?" in > >>> https://www.apache.org/legal/src-headers.html) > >>> > >>> Thank you, > >>> > >>> Vlad > >>> > >>> On Feb 23, 2025, at 4:37 PM, Dongjoon Hyun > >>> <dongjoon.h...@gmail.com<mailto:dongjoon.h...@gmail.com>> wrote: > >>> > >>> Please vote on releasing the following candidate as Apache Spark version > >>> 3.5.5. > >>> > >>> The vote is open until February 27th 1AM (PST) and passes if a majority > >>> +1 PMC votes are cast, with a minimum of 3 +1 votes. > >>> > >>> [ ] +1 Release this package as Apache Spark 3.5.5 > >>> [ ] -1 Do not release this package because ... > >>> > >>> To learn more about Apache Spark, please see https://spark.apache.org/ > >>> > >>> The tag to be voted on is v3.5.5-rc1 (commit > >>> 7c29c664cdc9321205a98a14858aaf8daaa19db2) > >>> https://github.com/apache/spark/tree/v3.5.5-rc1 > >>> > >>> The release files, including signatures, digests, etc. can be found at: > >>> https://dist.apache.org/repos/dist/dev/spark/v3.5.5-rc1-bin/ > >>> > >>> Signatures used for Spark RCs can be found in this file: > >>> https://dist.apache.org/repos/dist/dev/spark/KEYS > >>> > >>> The staging repository for this release can be found at: > >>> https://repository.apache.org/content/repositories/orgapachespark-1476/ > >>> > >>> The documentation corresponding to this release can be found at: > >>> https://dist.apache.org/repos/dist/dev/spark/v3.5.5-rc1-docs/ > >>> > >>> The list of bug fixes going into 3.5.5 can be found at the following URL: > >>> https://issues.apache.org/jira/projects/SPARK/versions/12355518 > >>> > >>> This release is using the release script of the tag v3.5.5-rc1. > >>> > >>> FAQ > >>> > >>> ========================= > >>> How can I help test this release? > >>> ========================= > >>> > >>> If you are a Spark user, you can help us test this release by taking > >>> an existing Spark workload and running on this release candidate, then > >>> reporting any regressions. > >>> > >>> If you're working in PySpark you can set up a virtual env and install > >>> the current RC and see if anything important breaks, in the Java/Scala > >>> you can add the staging repository to your projects resolvers and test > >>> with the RC (make sure to clean up the artifact cache before/after so > >>> you don't end up building with a out of date RC going forward). > >>> > >>> =========================================== > >>> What should happen to JIRA tickets still targeting 3.5.5? > >>> =========================================== > >>> > >>> The current list of open tickets targeted at 3.5.5 can be found at: > >>> https://issues.apache.org/jira/projects/SPARK and search for "Target > >>> Version/s" = 3.5.5 > >>> > >>> Committers should look at those and triage. Extremely important bug > >>> fixes, documentation, and API tweaks that impact compatibility should > >>> be worked on immediately. Everything else please retarget to an > >>> appropriate release. > >>> > >>> ================== > >>> But my bug isn't fixed? > >>> ================== > >>> > >>> In order to make timely releases, we will typically not hold the > >>> release unless the bug in question is a regression from the previous > >>> release. That being said, if there is something which is a regression > >>> that has not been correctly targeted please ping me or a committer to > >>> help target the issue. > >>> > >>> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe e-mail: > >> dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org> > >> > >> > > > > --------------------------------------------------------------------- > > To unsubscribe e-mail: > > dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org> > > > > > --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
