stankiewicz opened a new pull request, #40450:
URL: https://github.com/apache/beam/pull/40450
### Summary
`BeamModulePlugin.groovy` manually constructs the `<dependencies>` section
of published Maven POMs in `pom.withXml` (rather than using `from
components.java` due to Beam's shading model) and previously only exported
`shadow`/`implementation` (`compile` scope) and `provided` (`provided` scope).
As a result, `runtimeOnly` dependencies in unshaded modules were silently
omitted from published POMs, requiring either `implementation` +
`permitUnusedDeclared` workarounds or forcing downstream Maven users to
manually redeclare runtime dependencies.
Simply calling `generateDependenciesFromConfiguration(configuration:
'runtimeOnly', scope: 'runtime')` had three blockers:
1. **Shaded modules (`shadowClosure`):** Shaded modules bundle
`runtimeClasspath` into `shadowJar`, so exporting `runtimeOnly` would leak
shaded dependencies into the POM.
2. **Configuration inheritance (`runtimeOnly.extendsFrom(provided)`):**
Iterating `runtimeOnly.allDependencies` without deduplication duplicates every
`provided` dependency (and BOM) under `<scope>runtime</scope>`.
3. **Unpublished `ProjectDependency` targets & Configure-on-Demand:** Some
modules declare `runtimeOnly` dependencies on internal subprojects with
`publish: false` (`includeInJavaBom == false`), and running
`generatePomFileForMavenJavaPublication` with
`org.gradle.configureondemand=true` leaves `ProjectDependency` target
subprojects unconfigured unless evaluated before POM generation.
### Changes in `BeamModulePlugin.groovy`
1. **Unshaded `runtimeOnly` export:** Export `runtimeOnly` with
`<scope>runtime</scope>` only when `!configuration.shadowClosure`.
2. **Cross-scope and BOM deduplication:** Track emitted `[groupId,
artifactId, classifiers]` coordinates (`addedDependencies`) and BOM `[group,
name]` coordinates (`addedBoms`) across `compile`, `provided`, and `runtime`
scopes so `provided` or `compile` dependencies and BOMs are never duplicated in
`<scope>runtime</scope>`.
3. **Unpublished `ProjectDependency` filtering & Configure-on-Demand
safety:**
- Skip `ProjectDependency` entries where
`dependencyProject.ext.has('includeInJavaBom') &&
!dependencyProject.ext.includeInJavaBom` (`publish: false`).
- Attach a lightweight `evaluateForPom` task dependency to
`generatePomFileForMavenJavaPublication` for `ProjectDependency` targets in
published configurations so they are configured on demand without triggering
circular `evaluationDependsOn` errors.
### Impact on Published POMs (`2.78.0-SNAPSHOT` diff audit)
- **`beam-sdks-java-io-iceberg` (`:sdks:java:io:iceberg`):**
- Now exports its 8 `runtimeOnly` dependencies under
`<scope>runtime</scope>`: `iceberg-gcp`, `iceberg-aws`, `iceberg-aws-bundle`,
`iceberg-azure`, `iceberg-azure-bundle`, `gcs-connector`, `util-hadoop`, and
`hadoop-client`.
- `org.immutables:value` remains once under `<scope>provided</scope>` (no
duplication).
- **`beam-runners-google-cloud-dataflow-java`
(`:runners:google-cloud-dataflow-java`):**
- Allows `opentelemetry-exporter-otlp`,
`opentelemetry-sdk-extension-autoconfigure`, and
`beam-sdks-java-extensions-opentelemetry-gcp-auth-extension` to be declared as
`runtimeOnly` (exported under `<scope>runtime</scope>`) without
`implementation` + `permitUnusedDeclared` workarounds.
- `beam-sdks-java-io-kafka` remains once under `<scope>provided</scope>`,
and `com.google.cloud:libraries-bom` appears once in `<dependencyManagement>`.
- **`beam-sdks-java-extensions-sql-iceberg`
(`:sdks:java:extensions:sql:iceberg`):**
- **No POM diff:** Unpublished `runtimeOnly` project dependencies
(`:sdks:java:io:iceberg:bqms` and `:sdks:java:io:iceberg:hive`, which have
`publish: false`) are automatically excluded from the POM.
- **Shaded modules (e.g., `beam-sdks-java-core`, expansion service fat JARs,
Dataflow worker):**
- **No POM diff:** Guarded by `!configuration.shadowClosure`.
- **Other unshaded published modules declaring `runtimeOnly`:**
- `beam-sdks-java-io-delta`: Exports `org.apache.hadoop:hadoop-aws` under
`<scope>runtime</scope>`.
- `beam-sdks-java-io-pulsar`: Exports `org.apache.pulsar:pulsar-client`
and `org.apache.pulsar:pulsar-client-admin` under `<scope>runtime</scope>`.
- `beam-sdks-java-io-azure` & `beam-sdks-java-io-azure-cosmos`: Export
`io.netty:netty-all` under `<scope>runtime</scope>`.
- `beam-sdks-java-expansion-service`: Exports `com.github.jbellis:jamm`
and `org.slf4j:slf4j-jdk14` under `<scope>runtime</scope>`.
- `beam-examples-java`: Exports `beam-runners-direct-java`,
`hadoop-client`, and `gcs-connector` under `<scope>runtime</scope>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
