dejan2609 commented on code in PR #19513:
URL: https://github.com/apache/kafka/pull/19513#discussion_r2336213537
##########
.github/workflows/build.yml:
##########
@@ -164,7 +164,7 @@ jobs:
# Check if there are any empty files under ./site-docs/generated, If
any empty files are found, print an error
# message and list the empty files
run: |
- tar zxvf core/build/distributions/kafka_2.13-$(./gradlew properties
| grep version: | awk '{print $NF}' | head -n 1)-site-docs.tgz
+ tar zxvf distribution/build/distributions/kafka_2.13-$(./gradlew
properties | grep version: | awk '{print $NF}' | head -n 1)-site-docs.tgz
Review Comment:
Done, will push rebased commit(s) with distributions created in old (i.e.
`:core`) paths and then will return here for a full explanation.
##########
build.gradle:
##########
@@ -330,6 +329,19 @@ subprojects {
tasks.register('uploadArchives').configure { dependsOn(publish) }
}
+ def submodulesToIncludeInDistribution = ['core', 'tools', 'trogdor',
'shell', 'api', 'runtime', 'transforms',
+ 'json', 'file',
'basic-auth-extension', 'mirror', 'mirror-client',
+ 'streams', 'streams-scala',
'test-utils', 'examples', 'tools-api']
+
Review Comment:
Surprisingly enough: `project.name` for a leaf submodules resolves to a
submodule name only, but it seems that we are on a safe side (i.e. submodule
leaf names are unique):
https://stackoverflow.com/questions/76320016/is-it-possible-for-2-submodules-to-have-different-paths-but-identical-names/76342122#76342122
##########
.github/workflows/build.yml:
##########
@@ -164,7 +164,7 @@ jobs:
# Check if there are any empty files under ./site-docs/generated, If
any empty files are found, print an error
# message and list the empty files
run: |
- tar zxvf core/build/distributions/kafka_2.13-$(./gradlew properties
| grep version: | awk '{print $NF}' | head -n 1)-site-docs.tgz
+ tar zxvf distribution/build/distributions/kafka_2.13-$(./gradlew
properties | grep version: | awk '{print $NF}' | head -n 1)-site-docs.tgz
Review Comment:
Good news @chia7712 ! I have a solution that keeps distribution paths
as-is.
Just needs some testing/double-checking, hopefully will push it tomorrow.
##########
gradle/dependencies.gradle:
##########
@@ -119,13 +119,13 @@ versions += [
// When updating the scalafmt version please also update the version field
in checkstyle/.scalafmt.conf. scalafmt now
// has the version field as mandatory in its configuration, see
// https://github.com/scalameta/scalafmt/releases/tag/v3.1.0.
- scalafmt: "3.7.14",
- scoverage: "2.0.11",
+ scalafmt: "3.9.9",
Review Comment:
No, not at all.
Just to more precise: both Scala Coverage and Scala formatter are not
required for Gradle upgrade and can be left out.
##########
build.gradle:
##########
@@ -3904,6 +3856,109 @@ project(':connect:test-plugins') {
}
}
+project(':distribution') {
Review Comment:
Done, dummy folder `distribution` is removed.
##########
build.gradle:
##########
@@ -3904,6 +3856,109 @@ project(':connect:test-plugins') {
}
}
+project(':distribution') {
Review Comment:
@chia7712 here it is: #20566
##########
build.gradle:
##########
@@ -29,22 +29,21 @@ buildscript {
}
plugins {
- id 'com.github.ben-manes.versions' version '0.48.0'
+ id 'com.github.ben-manes.versions' version '0.52.0'
id 'idea'
id 'jacoco'
id 'java-library'
- id 'org.owasp.dependencycheck' version '8.2.1'
+ id 'org.owasp.dependencycheck' version '12.1.3'
id 'org.nosphere.apache.rat' version "0.8.1"
id "io.swagger.core.v3.swagger-gradle-plugin" version "${swaggerVersion}"
- id "com.github.spotbugs" version '6.2.3' apply false
+ id "com.github.spotbugs" version '6.2.5' apply false
id 'org.scoverage' version '8.0.3' apply false
- id 'com.gradleup.shadow' version '8.3.6' apply false
- id 'com.diffplug.spotless' version "6.25.0"
+ id 'com.gradleup.shadow' version '9.0.2' apply false
Review Comment:
Resolving conversation.
##########
gradlew:
##########
@@ -84,7 +86,7 @@ done
# shellcheck disable=SC2034
APP_BASE_NAME=${0##*/}
# Discard cd standard output in case $CDPATH is set
(https://github.com/gradle/gradle/issues/25036)
-APP_HOME=$( cd "${APP_HOME:-./}" > /dev/null && pwd -P ) || exit
+APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s\n' "$PWD" ) ||
exit
Review Comment:
No, they were not generated by Gradle 9 (but here is the twist: they are
generated by Gradle 8).
This change resolves the `gradlew` issue at the Gradle 8 level (basically:
`gradlew` content has diverged from the proper content for quite some time, and
this commit solves those issues).
Strangely enough: Gradle builds via Gradle wrapper still work (obviously) in
trunk (with Gradle 8) even without this change, but it is/was only a question
of time when this would escalate (and while working on Gradle upgrade from 8 to
9 I managed to discover it and solve it all along).
That's why I included this commit in this PR as the first one (in front of
Gradle 8.14.3 to 9.0.0 upgrade, that is):
- related Jira ticket: KAFKA-19591
- git commit (that can be merged into trunk separately, i.e. in isolation of
Gradle 8-->> 9 upgrade):
https://github.com/apache/kafka/pull/19513/commits/79f04159d96365d7b9c43c409134b1b1710e8c27
##########
build.gradle:
##########
@@ -3905,6 +3856,109 @@ project(':connect:test-plugins') {
}
}
+project(':distribution') {
+ base {
+ archivesName = "kafka_${versions.baseScala}"
+ }
+
+ configurations {
+ includeIntoDistribution
+ }
+
+ dependencies {
+ includeIntoDistribution project(path: ':core', configuration:
'releaseOnly')
+ includeIntoDistribution project(path: ':core', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':tools', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':trogdor', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':shell', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:api', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:runtime', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:transforms',
configuration: 'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:json', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:file', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:basic-auth-extension',
configuration: 'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:mirror', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:mirror-client',
configuration: 'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':streams', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':streams:streams-scala',
configuration: 'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':streams:test-utils',
configuration: 'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':streams:examples', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':tools:tools-api', configuration:
'resolvedRuntimeClasspath')
+ }
+
+ task siteDocsTar(dependsOn: [':core:genProtocolErrorDocs',
':core:genProtocolTypesDocs', ':core:genProtocolApiKeyDocs',
':core:genProtocolMessageDocs',
+ ':core:genAdminClientConfigDocs',
':core:genProducerConfigDocs', ':core:genConsumerConfigDocs',
+ ':core:genKafkaConfigDocs',
':core:genTopicConfigDocs', ':core:genGroupConfigDocs',
+ ':connect:runtime:genConnectConfigDocs',
':connect:runtime:genConnectTransformationDocs',
+ ':connect:runtime:genConnectPredicateDocs',
+ ':connect:runtime:genSinkConnectorConfigDocs',
':connect:runtime:genSourceConnectorConfigDocs',
+ ':streams:genStreamsConfigDocs',
':core:genConsumerMetricsDocs', ':core:genProducerMetricsDocs',
+ ':connect:runtime:genConnectMetricsDocs',
':connect:runtime:genConnectOpenAPIDocs',
+ ':connect:mirror:genMirrorSourceConfigDocs',
':connect:mirror:genMirrorCheckpointConfigDocs',
+ ':connect:mirror:genMirrorHeartbeatConfigDocs',
':connect:mirror:genMirrorConnectorConfigDocs',
+ ':storage:genRemoteLogManagerConfigDoc',
':storage:genRemoteLogMetadataManagerConfigDoc'], type: Tar) {
+ archiveClassifier = 'site-docs'
+ compression = Compression.GZIP
+ from project.file("$rootDir/docs")
+ into 'site-docs'
+ destinationDirectory =
project(':core').layout.buildDirectory.dir('distributions')
+ duplicatesStrategy 'exclude'
+ preserveFileTimestamps = true
+ }
+
+ tasks.create(name: "releaseTarGz", dependsOn:
project(':core').configurations.archives.artifacts, type: Tar) {
+ into "kafka_${versions.baseScala}-${archiveVersion.get()}"
+ compression = Compression.GZIP
+ from(project.file("$rootDir/bin")) { into "bin/" }
+ from(project.file("$rootDir/config")) { into "config/" }
+ from(project.file("$rootDir/licenses")) { into "licenses/" }
+ from "$rootDir/LICENSE-binary" rename {String filename ->
filename.replace("-binary", "")}
+ from "$rootDir/NOTICE-binary" rename {String filename ->
filename.replace("-binary", "")}
+ from(project.siteDocsTar) { into("site-docs/") }
+ from(configurations.includeIntoDistribution) { into("libs/") }
+ from(project(':core').configurations.archives.artifacts.files) {
into("libs/") }
+ from(project(':tools').jar) { into("libs/") }
+ from(project(':trogdor').jar) { into("libs/") }
+ from(project(':shell').jar) { into("libs/") }
+ from(project(':connect:api').jar) { into("libs/") }
+ from(project(':connect:runtime').jar) { into("libs/") }
+ from(project(':connect:transforms').jar) { into("libs/") }
+ from(project(':connect:json').jar) { into("libs/") }
+ from(project(':connect:file').jar) { into("libs/") }
+ from(project(':connect:basic-auth-extension').jar) { into("libs/") }
+ from(project(':connect:mirror').jar) { into("libs/") }
+ from(project(':connect:mirror-client').jar) { into("libs/") }
+ from(project(':streams').jar) { into("libs/") }
+ from(project(':streams:streams-scala').jar) { into("libs/") }
+ from(project(':streams:test-utils').jar) { into("libs/") }
+ from(project(':streams:examples').jar) { into("libs/") }
+ from(project(':tools:tools-api').jar) { into("libs/") }
+ destinationDirectory =
project(':core').layout.buildDirectory.dir('distributions')
Review Comment:
Resolved.
##########
gradle/dependencies.gradle:
##########
@@ -61,8 +61,8 @@ versions += [
commonsLang: "3.18.0",
commonsValidator: "1.9.0",
classgraph: "4.8.179",
- gradle: "8.14.3",
- grgit: "4.1.1",
+ gradle: "9.1.0",
+ grgit: "5.3.0",
Review Comment:
Done, commit is pushed.
##########
gradle/dependencies.gradle:
##########
@@ -61,8 +61,8 @@ versions += [
commonsLang: "3.18.0",
commonsValidator: "1.9.0",
classgraph: "4.8.179",
- gradle: "8.14.3",
- grgit: "4.1.1",
+ gradle: "9.1.0",
+ grgit: "5.3.0",
Review Comment:
No, the update for `grgit`is certainly not required here. I will remove that
and push is separately (as a MINOR PR).
##########
.github/workflows/build.yml:
##########
@@ -164,7 +164,7 @@ jobs:
# Check if there are any empty files under ./site-docs/generated, If
any empty files are found, print an error
# message and list the empty files
run: |
- tar zxvf core/build/distributions/kafka_2.13-$(./gradlew properties
| grep version: | awk '{print $NF}' | head -n 1)-site-docs.tgz
+ tar zxvf distribution/build/distributions/kafka_2.13-$(./gradlew
properties | grep version: | awk '{print $NF}' | head -n 1)-site-docs.tgz
Review Comment:
So, Gradle 9 is very strict with submodule boundaries and hence simple
version upgrade (from 8.14.3 to 9.0.0 for Gradle and from 8.3.6 to 9.0.2 for
shadow plugin) is not enough because `./gradlew releaseTarGz` yields an error:
```
> Task :core:releaseTarGz FAILED
[Incubating] Problems report is available at:
file:///home/dejan/kafka/build/reports/problems/problems-report.html
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':core:releaseTarGz'.
> Resolution of the configuration ':tools:tools-api:runtimeClasspath' was
attempted without an exclusive lock. This is unsafe and not allowed.
* Try:
> For more information, please refer to
https://docs.gradle.org/9.0.0/userguide/viewing_debugging_dependencies.html#sub:resolving-unsafe-configuration-resolution-errors
in the Gradle documentation.
```
So, in order to bypass this I opted to add these things in order to trick
Gradle 9 (so to say):
- (dummy) submodule `:distribution` and
- resolvable configuration `resolvedRuntimeClasspath` for required submodules
Initial solution included changed paths for generated kafka*.tgz files, but
now I managed to put them back into a `:core` submodule.
##########
gradle/dependencies.gradle:
##########
@@ -119,13 +119,13 @@ versions += [
// When updating the scalafmt version please also update the version field
in checkstyle/.scalafmt.conf. scalafmt now
// has the version field as mandatory in its configuration, see
// https://github.com/scalameta/scalafmt/releases/tag/v3.1.0.
- scalafmt: "3.7.14",
- scoverage: "2.0.11",
+ scalafmt: "3.9.9",
Review Comment:
Resolving conversation.
##########
gradle/dependencies.gradle:
##########
@@ -119,13 +119,13 @@ versions += [
// When updating the scalafmt version please also update the version field
in checkstyle/.scalafmt.conf. scalafmt now
// has the version field as mandatory in its configuration, see
// https://github.com/scalameta/scalafmt/releases/tag/v3.1.0.
- scalafmt: "3.7.14",
- scoverage: "2.0.11",
+ scalafmt: "3.9.9",
Review Comment:
Done, removed in rebased commits.
##########
.github/workflows/build.yml:
##########
@@ -164,7 +164,7 @@ jobs:
# Check if there are any empty files under ./site-docs/generated, If
any empty files are found, print an error
# message and list the empty files
run: |
- tar zxvf core/build/distributions/kafka_2.13-$(./gradlew properties
| grep version: | awk '{print $NF}' | head -n 1)-site-docs.tgz
+ tar zxvf distribution/build/distributions/kafka_2.13-$(./gradlew
properties | grep version: | awk '{print $NF}' | head -n 1)-site-docs.tgz
Review Comment:
Resolving conversation (paths for distribution .tgz files are intact).
##########
.github/actions/setup-gradle/action.yml:
##########
@@ -42,7 +42,7 @@ runs:
distribution: temurin
java-version: ${{ inputs.java-version }}
- name: Setup Gradle
- uses:
gradle/actions/setup-gradle@94baf225fe0a508e581a564467443d0e2379123b # v4.3.0
Review Comment:
Done, github action version is changed as suggested.
##########
build.gradle:
##########
@@ -3904,6 +3856,109 @@ project(':connect:test-plugins') {
}
}
+project(':distribution') {
Review Comment:
I may have some good news: it seems that we can avoid this folder
:crossed_fingers:
While it's true that Gradle 9+ fails with an error if subprojects are
missing their own folder (see below) that behavior can be easily avoided:
https://docs.gradle.org/9.1.0/userguide/multi_project_builds.html#include_existing_projects_only
Will commit in no time.
```
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git log -2 --oneline
218f5a941a (HEAD -> KAFKA-19174, origin/KAFKA-19174) KAFKA-19174 Gradle
version upgrade: 9.0.0 -->> 9.1.0
f0946dbc6a KAFKA-19654 do not fail Github Actions flaky/new build matrix
build when no test are discovered (i.e. solution for KAFKA-16801 is expanded)
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git status
On branch KAFKA-19174
Your branch is up-to-date with 'origin/KAFKA-19174'.
Changes to be committed:
(use "git restore --staged <file>..." to unstage)
deleted: distribution/.gitkeep
dejan@dejan-HP-ProBook-450-G7:~/kafka$ ./gradlew clean releaseTarGz -q
FAILURE: Build failed with an exception.
* What went wrong:
Configuring project ':distribution' without an existing directory is not
allowed. The configured projectDirectory '/home/dejan/kafka/distribution' does
not exist, can't be written to or is not a directory.
* Try:
> Make sure the project directory exists and is writable.
BUILD FAILED in 1s
dejan@dejan-HP-ProBook-450-G7:~/kafka$
```
##########
build.gradle:
##########
@@ -3904,6 +3856,109 @@ project(':connect:test-plugins') {
}
}
+project(':distribution') {
Review Comment:
On the other hand: Gradle submodule `distribution` is required for build to
work with Gradle 9.
I will provide an in-depth explanation over the weekend.
##########
build.gradle:
##########
@@ -29,22 +29,21 @@ buildscript {
}
plugins {
- id 'com.github.ben-manes.versions' version '0.48.0'
+ id 'com.github.ben-manes.versions' version '0.52.0'
id 'idea'
id 'jacoco'
id 'java-library'
- id 'org.owasp.dependencycheck' version '8.2.1'
+ id 'org.owasp.dependencycheck' version '12.1.3'
id 'org.nosphere.apache.rat' version "0.8.1"
id "io.swagger.core.v3.swagger-gradle-plugin" version "${swaggerVersion}"
- id "com.github.spotbugs" version '6.2.3' apply false
+ id "com.github.spotbugs" version '6.2.5' apply false
id 'org.scoverage' version '8.0.3' apply false
- id 'com.gradleup.shadow' version '8.3.6' apply false
- id 'com.diffplug.spotless' version "6.25.0"
+ id 'com.gradleup.shadow' version '9.0.2' apply false
Review Comment:
Hi @Goooler
git
[commit](https://github.com/apache/kafka/commit/d067c6c04089a3d24e1f72e6cb1b10b0d85f76da)
you are referring to was merged into trunk yesterday, while this PR is not
synced with trunk for at least 5 days (and tests for
https://github.com/GradleUp/shadow/issues/1769 were done locally, not on Github
Actions CI - don't ask me why that specific case is not a part of CI test, that
is a different pair of shoes).
<img width="1024" height="454" alt="image"
src="https://github.com/user-attachments/assets/1baf67e2-27fc-4dbb-8a16-c6efa45fe931";
/>
So I can't revert that commit for sure in my PR (mind you: I'm not judging
your assessment about issues that specific commit may or may not impose, just
stating the obvious from a git revert POV).
Furthermore: Apache Kafka Gradle build is massive/overwhelming/overzealous,
and working on it takes a massive toll on members/maintainers (and simple
contributors like me share that sentiment).
Due to Gradle build being so complicated, it is really hard to upgrade
Gradle versions.
I do believe you that Shadow plugin 8.3.6 is compatible in (any other ?)
Gradle 9 setup but here that is not the case (build doesn't just breaks with
error, it bends space-time, so to say).
That is why I included (among all other changes related to Gradle 9),
upgrade for Shadow plugin to version 9 and things were working just fine until
we discovered some corner case (as it turns out: on Apache Kafka side) that is
not covered via CI.
I thought that it was a corner case on Shadow plugin side and hence I
created an issue. It seems I was wrong and I want to make amends (like a real
one, not in Git :cowboy_hat_face:).
I can only imagine how complicated things can be on a core side when this
implementation stuff can be so intermingled and sketchy.
Having said all of this: I will rewind all of this and try to come up with a
plan how/where to proceed.
Peace be with you.
##########
build.gradle:
##########
@@ -3904,6 +3856,109 @@ project(':connect:test-plugins') {
}
}
+project(':distribution') {
Review Comment:
Marking this as resolved.
##########
build.gradle:
##########
@@ -29,22 +29,21 @@ buildscript {
}
plugins {
- id 'com.github.ben-manes.versions' version '0.48.0'
+ id 'com.github.ben-manes.versions' version '0.52.0'
id 'idea'
id 'jacoco'
id 'java-library'
- id 'org.owasp.dependencycheck' version '8.2.1'
+ id 'org.owasp.dependencycheck' version '12.1.3'
id 'org.nosphere.apache.rat' version "0.8.1"
id "io.swagger.core.v3.swagger-gradle-plugin" version "${swaggerVersion}"
- id "com.github.spotbugs" version '6.2.3' apply false
+ id "com.github.spotbugs" version '6.2.5' apply false
id 'org.scoverage' version '8.0.3' apply false
- id 'com.gradleup.shadow' version '8.3.6' apply false
- id 'com.diffplug.spotless' version "6.25.0"
+ id 'com.gradleup.shadow' version '9.0.2' apply false
Review Comment:
I'll give it a try.
##########
build.gradle:
##########
@@ -29,22 +29,21 @@ buildscript {
}
plugins {
- id 'com.github.ben-manes.versions' version '0.48.0'
+ id 'com.github.ben-manes.versions' version '0.52.0'
id 'idea'
id 'jacoco'
id 'java-library'
- id 'org.owasp.dependencycheck' version '8.2.1'
+ id 'org.owasp.dependencycheck' version '12.1.3'
id 'org.nosphere.apache.rat' version "0.8.1"
id "io.swagger.core.v3.swagger-gradle-plugin" version "${swaggerVersion}"
- id "com.github.spotbugs" version '6.2.3' apply false
+ id "com.github.spotbugs" version '6.2.5' apply false
id 'org.scoverage' version '8.0.3' apply false
- id 'com.gradleup.shadow' version '8.3.6' apply false
- id 'com.diffplug.spotless' version "6.25.0"
+ id 'com.gradleup.shadow' version '9.0.2' apply false
Review Comment:
Nice catch, let me try to address this.
##########
.github/workflows/build.yml:
##########
@@ -164,7 +164,7 @@ jobs:
# Check if there are any empty files under ./site-docs/generated, If
any empty files are found, print an error
# message and list the empty files
run: |
- tar zxvf core/build/distributions/kafka_2.13-$(./gradlew properties
| grep version: | awk '{print $NF}' | head -n 1)-site-docs.tgz
+ tar zxvf distribution/build/distributions/kafka_2.13-$(./gradlew
properties | grep version: | awk '{print $NF}' | head -n 1)-site-docs.tgz
Review Comment:
Yes, this is a sensitive issue.
I may have an alternative way (that keeps paths as-is); will try to provide
different approach and explain rationale for both solutions.
##########
build.gradle:
##########
@@ -3905,6 +3856,109 @@ project(':connect:test-plugins') {
}
}
+project(':distribution') {
+ base {
+ archivesName = "kafka_${versions.baseScala}"
+ }
+
+ configurations {
+ includeIntoDistribution
+ }
+
+ dependencies {
+ includeIntoDistribution project(path: ':core', configuration:
'releaseOnly')
+ includeIntoDistribution project(path: ':core', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':tools', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':trogdor', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':shell', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:api', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:runtime', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:transforms',
configuration: 'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:json', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:file', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:basic-auth-extension',
configuration: 'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:mirror', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':connect:mirror-client',
configuration: 'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':streams', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':streams:streams-scala',
configuration: 'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':streams:test-utils',
configuration: 'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':streams:examples', configuration:
'resolvedRuntimeClasspath')
+ includeIntoDistribution project(path: ':tools:tools-api', configuration:
'resolvedRuntimeClasspath')
+ }
+
+ task siteDocsTar(dependsOn: [':core:genProtocolErrorDocs',
':core:genProtocolTypesDocs', ':core:genProtocolApiKeyDocs',
':core:genProtocolMessageDocs',
+ ':core:genAdminClientConfigDocs',
':core:genProducerConfigDocs', ':core:genConsumerConfigDocs',
+ ':core:genKafkaConfigDocs',
':core:genTopicConfigDocs', ':core:genGroupConfigDocs',
+ ':connect:runtime:genConnectConfigDocs',
':connect:runtime:genConnectTransformationDocs',
+ ':connect:runtime:genConnectPredicateDocs',
+ ':connect:runtime:genSinkConnectorConfigDocs',
':connect:runtime:genSourceConnectorConfigDocs',
+ ':streams:genStreamsConfigDocs',
':core:genConsumerMetricsDocs', ':core:genProducerMetricsDocs',
+ ':connect:runtime:genConnectMetricsDocs',
':connect:runtime:genConnectOpenAPIDocs',
+ ':connect:mirror:genMirrorSourceConfigDocs',
':connect:mirror:genMirrorCheckpointConfigDocs',
+ ':connect:mirror:genMirrorHeartbeatConfigDocs',
':connect:mirror:genMirrorConnectorConfigDocs',
+ ':storage:genRemoteLogManagerConfigDoc',
':storage:genRemoteLogMetadataManagerConfigDoc'], type: Tar) {
+ archiveClassifier = 'site-docs'
+ compression = Compression.GZIP
+ from project.file("$rootDir/docs")
+ into 'site-docs'
+ destinationDirectory =
project(':core').layout.buildDirectory.dir('distributions')
+ duplicatesStrategy 'exclude'
+ preserveFileTimestamps = true
+ }
+
+ tasks.create(name: "releaseTarGz", dependsOn:
project(':core').configurations.archives.artifacts, type: Tar) {
+ into "kafka_${versions.baseScala}-${archiveVersion.get()}"
+ compression = Compression.GZIP
+ from(project.file("$rootDir/bin")) { into "bin/" }
+ from(project.file("$rootDir/config")) { into "config/" }
+ from(project.file("$rootDir/licenses")) { into "licenses/" }
+ from "$rootDir/LICENSE-binary" rename {String filename ->
filename.replace("-binary", "")}
+ from "$rootDir/NOTICE-binary" rename {String filename ->
filename.replace("-binary", "")}
+ from(project.siteDocsTar) { into("site-docs/") }
+ from(configurations.includeIntoDistribution) { into("libs/") }
+ from(project(':core').configurations.archives.artifacts.files) {
into("libs/") }
+ from(project(':tools').jar) { into("libs/") }
+ from(project(':trogdor').jar) { into("libs/") }
+ from(project(':shell').jar) { into("libs/") }
+ from(project(':connect:api').jar) { into("libs/") }
+ from(project(':connect:runtime').jar) { into("libs/") }
+ from(project(':connect:transforms').jar) { into("libs/") }
+ from(project(':connect:json').jar) { into("libs/") }
+ from(project(':connect:file').jar) { into("libs/") }
+ from(project(':connect:basic-auth-extension').jar) { into("libs/") }
+ from(project(':connect:mirror').jar) { into("libs/") }
+ from(project(':connect:mirror-client').jar) { into("libs/") }
+ from(project(':streams').jar) { into("libs/") }
+ from(project(':streams:streams-scala').jar) { into("libs/") }
+ from(project(':streams:test-utils').jar) { into("libs/") }
+ from(project(':streams:examples').jar) { into("libs/") }
+ from(project(':tools:tools-api').jar) { into("libs/") }
+ destinationDirectory =
project(':core').layout.buildDirectory.dir('distributions')
Review Comment:
Done, commit (small comment) is pushed.
##########
build.gradle:
##########
@@ -29,22 +29,21 @@ buildscript {
}
plugins {
- id 'com.github.ben-manes.versions' version '0.48.0'
+ id 'com.github.ben-manes.versions' version '0.52.0'
id 'idea'
id 'jacoco'
id 'java-library'
- id 'org.owasp.dependencycheck' version '8.2.1'
+ id 'org.owasp.dependencycheck' version '12.1.3'
id 'org.nosphere.apache.rat' version "0.8.1"
id "io.swagger.core.v3.swagger-gradle-plugin" version "${swaggerVersion}"
- id "com.github.spotbugs" version '6.2.3' apply false
+ id "com.github.spotbugs" version '6.2.5' apply false
id 'org.scoverage' version '8.0.3' apply false
- id 'com.gradleup.shadow' version '8.3.6' apply false
- id 'com.diffplug.spotless' version "6.25.0"
+ id 'com.gradleup.shadow' version '9.0.2' apply false
Review Comment:
Resolving conversation, action points from above will be added into JIRA
ticket(s).
##########
build.gradle:
##########
@@ -29,22 +29,21 @@ buildscript {
}
plugins {
- id 'com.github.ben-manes.versions' version '0.48.0'
+ id 'com.github.ben-manes.versions' version '0.52.0'
id 'idea'
id 'jacoco'
id 'java-library'
- id 'org.owasp.dependencycheck' version '8.2.1'
+ id 'org.owasp.dependencycheck' version '12.1.3'
id 'org.nosphere.apache.rat' version "0.8.1"
id "io.swagger.core.v3.swagger-gradle-plugin" version "${swaggerVersion}"
- id "com.github.spotbugs" version '6.2.3' apply false
+ id "com.github.spotbugs" version '6.2.5' apply false
id 'org.scoverage' version '8.0.3' apply false
- id 'com.gradleup.shadow' version '8.3.6' apply false
- id 'com.diffplug.spotless' version "6.25.0"
+ id 'com.gradleup.shadow' version '9.0.2' apply false
Review Comment:
It seems like a shadow plugin version 8.3.9 **_is_** a way to go :rocket:
Action points for me:
- update that closed shadow issue for future readers (it would be much
easier to sum it up there - this PR is packed with comments)
- Create a separate Kafka jira ticket for a Shadow plugin upgrade (to
version 9+, at some point in the future)
- rebase this PR onto Kafka trunk branch and monitor how Gradle build
behaves after
https://github.com/apache/kafka/commit/d067c6c04089a3d24e1f72e6cb1b10b0d85f76da
commit is included
- explain to @chia7712 why the new `:distribution` submodule is required
(for Gradle build to work)
Last but not least: Thank you for helping us out with this @Goooler.
##########
gradlew:
##########
@@ -84,7 +86,7 @@ done
# shellcheck disable=SC2034
APP_BASE_NAME=${0##*/}
# Discard cd standard output in case $CDPATH is set
(https://github.com/gradle/gradle/issues/25036)
-APP_HOME=$( cd "${APP_HOME:-./}" > /dev/null && pwd -P ) || exit
+APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s\n' "$PWD" ) ||
exit
Review Comment:
Note: explanation was far too long for conversation so I added multiple
comments below.
I will keep this conversation open.
##########
build.gradle:
##########
@@ -1264,6 +1268,27 @@ project(':core') {
from(project(':streams:examples').jar) { into("libs/") }
from(project(':streams:examples').configurations.runtimeClasspath) {
into("libs/") }
duplicatesStrategy 'exclude'
+ preserveFileTimestamps = true
Review Comment:
Prologue (comment from back in August):
https://github.com/apache/kafka/pull/19513#issuecomment-3217444662
Related links:
- Gradle issue: https://github.com/gradle/gradle/issues/34643
- Gradle DSL:
https://docs.gradle.org/9.1.0/dsl/org.gradle.api.tasks.bundling.Tar.html#org.gradle.api.tasks.bundling.Tar:preserveFileTimestamps
I just tested again, with the following procedure: comment out
`preserveFileTimestamps = true` for tasks `:core:siteDocsTar` and
`:core:releaseTarGz` and execute `./gradlew clean releaseTarGz -i`:
```
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git log -3 --oneline
4278e5621e (HEAD -> KAFKA-19174, origin/KAFKA-19174) KAFKA-19174 Gradle
version upgrade (8 -->> 9); build logic is refactored
9318bd3480 KAFKA-19591 solving issues with `gradlew` content: an old
template version (i.e. file `unixStartScript.txt`) was being referenced
d76442e5a6 (origin/trunk, origin/HEAD, trunk) MINOR: Fix a minor copy&paste
typo in comments (#20634)
dejan@dejan-HP-ProBook-450-G7:~/kafka$ git diff
diff --git a/build.gradle b/build.gradle
index c3cc954e4b..8c7f1d19b4 100644
--- a/build.gradle
+++ b/build.gradle
@@ -1222,7 +1222,7 @@ project(':core') {
from project.file("$rootDir/docs")
into 'site-docs'
duplicatesStrategy 'exclude'
- preserveFileTimestamps = true
+ //preserveFileTimestamps = true
}
tasks.create(name: "releaseTarGz", dependsOn:
configurations.archives.artifacts, type: Tar) {
@@ -1268,7 +1268,7 @@ project(':core') {
from(project(':streams:examples').jar) { into("libs/") }
from(project(':streams:examples').configurations.runtimeClasspath) {
into("libs/") }
duplicatesStrategy 'exclude'
- preserveFileTimestamps = true
+ //preserveFileTimestamps = true
filePermissions {
user {
read = true
dejan@dejan-HP-ProBook-450-G7:~/kafka$ ./gradlew clean releaseTarGz -q
Starting build with version 4.2.0-SNAPSHOT (commit id 4278e562) using Gradle
9.1.0, Java 17 and Scala 2.13.16
Build properties: ignoreFailures=false, maxParallelForks=8,
maxScalacThreads=8, maxTestRetries=0
MessageGenerator: processed 4 Kafka message JSON file(s).
...
```
Unpack generated distributions:
- cd core/build/distributions/
- tar xvfz kafka_2.13-4.2.0-SNAPSHOT.tgz
- tar xvfz kafka_2.13-4.2.0-SNAPSHOT-site-docs.tgz
**Folders show the wrong date: 02nd Jan 1970:** (with default
`preserveFileTimestamps = false`) :arrow_down_small:
```
dejan@dejan-HP-ProBook-450-G7:~/kafka/core/build/distributions$ ls -la
total 136280
drwxrwxr-x 4 dejan dejan 4096 Oct 6 10:09 .
drwxrwxr-x 9 dejan dejan 4096 Oct 6 10:05 ..
drwxr-xr-x 7 dejan dejan 4096 Jan 2 1970 kafka_2.13-4.2.0-SNAPSHOT
-rw-rw-r-- 1 dejan dejan 3754359 Oct 6 10:05
kafka_2.13-4.2.0-SNAPSHOT-site-docs.tgz
-rw-rw-r-- 1 dejan dejan 135767889 Oct 6 10:06 kafka_2.13-4.2.0-SNAPSHOT.tgz
drwxr-xr-x 7 dejan dejan 4096 Jan 2 1970 site-docs
dejan@dejan-HP-ProBook-450-G7:~/kafka/core/build/distributions$
```
##########
wrapper.gradle:
##########
@@ -35,14 +35,12 @@ task bootstrapWrapper() {
def wrapperBasePath = "\$APP_HOME/gradle/wrapper"
def wrapperJarPath = wrapperBasePath + "/gradle-wrapper.jar"
- // Add a trailing zero to the version if needed.
Review Comment:
@chia7712 the thing is that Gradle team aligned/consolidated tag names and
version numbers starting with version 9: now all tag names/versions have two
dots (and three digits):
https://github.com/apache/kafka/pull/20327#pullrequestreview-3131743127
So, my intention was to remove redundant check (rationale: if all versions
are the same there is no need to add a trailing zero :slightly_smiling_face:
and I assume it is highly unlikely that Gradle team will revert this decision).
LMK what you think: should we leave as-is in this PR or you want to be extra
cautious so I can fall back to trunk for this check ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
