AFAIK, there are 3 ways to satisfy ASF source release policy: 1. Use source code to build test jars and remove jars from the source release and they may still be part of the source in the github repo (preferable). 2. Publish jars as artifacts in Maven central and pull them as a dependency. 3. Repackage jars to include LICENSE.
Thank you, Vlad On Feb 25, 2025, at 7:18 PM, Herman van Hovell <her...@databricks.com.INVALID> wrote: https://github.com/apache/spark/blob/master/sql/connect/client/jvm/src/test/resources/StubClassDummyUdf.scala (commit 408c3df) is the source code for the following jars: * ./connector/connect/server/src/test/resources/udf_noA.jar * ./connector/connect/client/jvm/src/test/resources/udf2.13.jar * ./connector/connect/client/jvm/src/test/resources/udf2.12.jar https://github.com/apache/spark/blob/master/sql/hive/src/test/resources/regression-test-SPARK-8489/Main.scala and https://github.com/apache/spark/blob/master/sql/hive/src/test/resources/regression-test-SPARK-8489/MyCoolClass.scala are the source files for the following jars: * ./sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.12.jar * ./sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.13.jar * ./sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar ./sql/core/src/test/resources/SPARK-33084.jar was added in https://github.com/apache/spark/commit/771c538620e66be2d0fb0e383e4aa37b4d29f7eb. The commit contains the source code. I will get the source code for the following jars checked in: * ./core/src/test/resources/TestHelloV3_2.12.jar * ./core/src/test/resources/TestHelloV2_2.12.jar * ./core/src/test/resources/TestHelloV2_2.13.jar * ./core/src/test/resources/TestHelloV3_2.13.jar * ./connector/connect/common/src/test/resources/artifact-tests/junitLargeJar.jar * ./connector/connect/common/src/test/resources/artifact-tests/smallJar.jar * ./connector/connect/client/jvm/src/test/resources/TestHelloV2_2.12.jar * ./connector/connect/client/jvm/src/test/resources/TestHelloV2_2.13.jar * ./data/artifact-tests/junitLargeJar.jar * ./data/artifact-tests/smallJar.jar On Tue, Feb 25, 2025 at 9:22 PM Rozov, Vlad <vro...@amazon.com.invalid> wrote: Right, the issue does not seem to be new for 3.5 and it is not new for 3.5.5. Here is the list of all jars I found in the source release: ./core/src/test/resources/TestHelloV3_2.12.jar ./core/src/test/resources/TestHelloV2_2.12.jar ./core/src/test/resources/TestHelloV2_2.13.jar ./core/src/test/resources/TestHelloV3_2.13.jar ./core/src/test/resources/TestUDTF.jar ./connector/connect/server/src/test/resources/udf_noA.jar ./connector/connect/common/src/test/resources/artifact-tests/junitLargeJar.jar ./connector/connect/common/src/test/resources/artifact-tests/smallJar.jar ./connector/connect/client/jvm/src/test/resources/TestHelloV2_2.12.jar ./connector/connect/client/jvm/src/test/resources/TestHelloV2_2.13.jar ./connector/connect/client/jvm/src/test/resources/udf2.13.jar ./connector/connect/client/jvm/src/test/resources/udf2.12.jar ./data/artifact-tests/junitLargeJar.jar ./data/artifact-tests/smallJar.jar ./sql/core/src/test/resources/SPARK-33084.jar ./sql/hive/src/test/noclasspath/hive-test-udfs.jar ./sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.12.jar ./sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.13.jar ./sql/hive/src/test/resources/regression-test-SPARK-8489/test-2.11.jar ./sql/hive/src/test/resources/SPARK-21101-1.0.jar ./sql/hive/src/test/resources/data/files/TestSerDe.jar ./sql/hive/src/test/resources/TestUDTF.jar ./sql/hive-thriftserver/src/test/resources/TestUDTF.jar Thank you, Vlad > On Feb 25, 2025, at 3:03 PM, Dongjoon Hyun > <dongj...@apache.org<mailto:dongj...@apache.org>> wrote: > > Just for the record, `TestUDTF.jar` test resource seems to be one of the > oldest examples which Vlad pointed. > > https://github.com/apache/spark/blob/v2.0.0/core/src/test/resources/TestUDTF.jar > (Added 9 years ago) > > $ tar tvf spark-2.0.0.tgz | grep TestUDTF.jar > -rw-r--r-- 0 jenkins jenkins 1328 Jul 19 2016 > spark-2.0.0/sql/hive/src/test/resources/TestUDTF.jar > -rw-r--r-- 0 jenkins jenkins 1328 Jul 19 2016 > spark-2.0.0/sql/hive-thriftserver/src/test/resources/TestUDTF.jar > -rw-r--r-- 0 jenkins jenkins 1328 Jul 19 2016 > spark-2.0.0/core/src/test/resources/TestUDTF.jar > > Dongjoon. > > > On 2025/02/25 22:47:07 Dongjoon Hyun wrote: >> Thank you for reply, Vlad. >> >> Do you mean SPARK-44246 violated ASF policy since Apache Spark 3.5.0 by >> injecting a unlicensed test jar, `TestHelloV2.jar`? >> >> https://issues.apache.org/jira/browse/SPARK-44246 >> https://github.com/apache/spark/pull/41789 >> >> And, it was spread to `TestHelloV3_2.13.jar` via SPARK-44297 in the same way? >> >> https://issues.apache.org/jira/browse/SPARK-44297 >> https://github.com/apache/spark/pull/41852 >> >> >> Dongjoon. >> >> On 2025/02/25 19:53:45 "Rozov, Vlad" wrote: >>> I am not sure if this was already discussed and noted, so want to confirm >>> with PMC members: >>> >>> I see several (test) JAR files included into the ASF source release that do >>> not have LICENSE in the MANIFEST or META-INF and do not have the source >>> code. For example core/src/test/resources/TestHelloV3_2.13.jar. AFAIK it >>> may be against ASF source release policy to include such jars into the >>> source release (see "Does the policy apply to binary/object files, such as >>> executables or JAR files?" in https://www.apache.org/legal/src-headers.html) >>> >>> Thank you, >>> >>> Vlad >>> >>> On Feb 23, 2025, at 4:37 PM, Dongjoon Hyun >>> <dongjoon.h...@gmail.com<mailto:dongjoon.h...@gmail.com>> wrote: >>> >>> Please vote on releasing the following candidate as Apache Spark version >>> 3.5.5. >>> >>> The vote is open until February 27th 1AM (PST) and passes if a majority +1 >>> PMC votes are cast, with a minimum of 3 +1 votes. >>> >>> [ ] +1 Release this package as Apache Spark 3.5.5 >>> [ ] -1 Do not release this package because ... >>> >>> To learn more about Apache Spark, please see https://spark.apache.org/ >>> >>> The tag to be voted on is v3.5.5-rc1 (commit >>> 7c29c664cdc9321205a98a14858aaf8daaa19db2) >>> https://github.com/apache/spark/tree/v3.5.5-rc1 >>> >>> The release files, including signatures, digests, etc. can be found at: >>> https://dist.apache.org/repos/dist/dev/spark/v3.5.5-rc1-bin/ >>> >>> Signatures used for Spark RCs can be found in this file: >>> https://dist.apache.org/repos/dist/dev/spark/KEYS >>> >>> The staging repository for this release can be found at: >>> https://repository.apache.org/content/repositories/orgapachespark-1476/ >>> >>> The documentation corresponding to this release can be found at: >>> https://dist.apache.org/repos/dist/dev/spark/v3.5.5-rc1-docs/ >>> >>> The list of bug fixes going into 3.5.5 can be found at the following URL: >>> https://issues.apache.org/jira/projects/SPARK/versions/12355518 >>> >>> This release is using the release script of the tag v3.5.5-rc1. >>> >>> FAQ >>> >>> ========================= >>> How can I help test this release? >>> ========================= >>> >>> If you are a Spark user, you can help us test this release by taking >>> an existing Spark workload and running on this release candidate, then >>> reporting any regressions. >>> >>> If you're working in PySpark you can set up a virtual env and install >>> the current RC and see if anything important breaks, in the Java/Scala >>> you can add the staging repository to your projects resolvers and test >>> with the RC (make sure to clean up the artifact cache before/after so >>> you don't end up building with a out of date RC going forward). >>> >>> =========================================== >>> What should happen to JIRA tickets still targeting 3.5.5? >>> =========================================== >>> >>> The current list of open tickets targeted at 3.5.5 can be found at: >>> https://issues.apache.org/jira/projects/SPARK and search for "Target >>> Version/s" = 3.5.5 >>> >>> Committers should look at those and triage. Extremely important bug >>> fixes, documentation, and API tweaks that impact compatibility should >>> be worked on immediately. Everything else please retarget to an >>> appropriate release. >>> >>> ================== >>> But my bug isn't fixed? >>> ================== >>> >>> In order to make timely releases, we will typically not hold the >>> release unless the bug in question is a regression from the previous >>> release. That being said, if there is something which is a regression >>> that has not been correctly targeted please ping me or a committer to >>> help target the issue. >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: >> dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org> >> >> > > --------------------------------------------------------------------- > To unsubscribe e-mail: > dev-unsubscr...@spark.apache.org<mailto:dev-unsubscr...@spark.apache.org> >
