pjfanning commented on issue #2043:
URL:
https://github.com/apache/pekko-connectors/issues/2043#issuecomment-6613494379
### Plan for the remaining items: one PR per connector group
### Context
This issue tracks Java API problems found in a review of every `javadsl`
package and the settings and model types those APIs reach. Six bugs already
have their own PRs: #2044 (BigQuery `withoutDeduplication`), #2045 (FCM
`light_off_duration`), #2046 (EventBridge `flowSeq`/`flowList`), #2047
(google-common `ForwardProxy.create`), #2048 (mqtt-streaming `Command`) and
#2049 (testkit doc). IronMQ is out of scope because it is being removed in
#1898.
This plan fixes the **remaining high, medium and low items** in 11 grouped
PRs.
### Approach
- **Scope:** high, medium and low items, excluding ironmq and anything
covered by #2044–#2049.
- **Compatibility:** add a Java-friendly method and deprecate the old one
(`@deprecated("…", "2.0.0")`). No binary breaks.
- **Generic type arguments only**
(`CompletionStage[Long]`→`[java.lang.Long]`,
`Optional[Boolean]`→`[java.lang.Boolean]`, `Flow[T, Boolean, _]`, `Pair[String,
Int]`): change in place and add `IncompatibleSignatureProblem` excludes. The
repo has `mimaReportSignatureProblems := true` (`project/Common.scala:49`).
Erased bytecode and boxed runtime values are unchanged.
- **Ambiguous lambda overloads** (Java functional interface vs
`scala.FunctionN` of the same arity): first prove the ambiguity with a Java
test that fails to compile. Then add a distinctly named Java method and
deprecate the clashing Java-function overload. Scala overloads stay.
### Conventions for every PR
- **Java getters:** follow the
`reference/src/main/scala/.../reference/settings.scala` pattern:
```scala
/** Java API … @since 2.0.0 */
def getX(): Optional[T] = x.toJava
```
Use `java.time.Duration` via `scala.jdk.DurationConverters`,
`java.util.List`/`Map`/`Set` via `scala.jdk.CollectionConverters`, and
`OptionalLong`/`OptionalInt` where natural. Match the module's existing `getX`
vs `getX()` style.
- **Java withers and factories:** add `withX(java.time.Duration)`,
`withX(java.util.List/Map)` and `create(...)` overloads beside the Scala ones.
- **Case objects needing `X$.MODULE$`:** add `def getInstance: X.type =
this`, following the existing `elasticsearch`
`StringMessageWriter.getInstance`. Top-level objects get static forwarders, so
Java calls `X.getInstance()`.
- **Symbolic operators** (`&&`, `||`, `unary_!` on `ConditionBuilder`): add
`and` / `or` / `not` aliases.
- **Scala default arguments and `implicit` lists in javadsl:** add explicit
overloads and keep the existing method.
- **Every new public API gets `@since 2.0.0`.**
- **Imports:** `import org.apache.pekko` then `pekko.`-relative imports.
- **License headers:** let sbt add them (`headerCreateAll`). New
`mima-filters/**.excludes` files get the hand-written ASF `#` header.
- **Excludes location:** the module's existing
`src/main/mima-filters/2.0.x.backward(s).excludes/` directory. The name varies
per module, so check before adding.
- **Commits and PRs:** AGENTS.md
Motivation/Modification/Result/Tests/References format with `Refs #2043`. Title
prefixed with the group's modules. No AI or `Co-authored-by` trailers.
### Tests (directional)
- Each touched module gets a **Java JUnit 5 (Jupiter) test** at
`src/test/java/org/apache/pekko/stream/connectors/<module>/javadsl/<Module>JavaApiTest.java`.
Jupiter is the repo norm, e.g.
`sqs/src/test/java/org/apache/pekko/.../javadsl/BaseSqsTest.java`. The test
calls the new methods from Java, which also proves overload resolution, erasure
and lambda use compile. It uses unit-level assertions on settings and model
types and needs no broker.
- Stage-level behaviour uses existing mock specs where they exist (e.g. the
EventBridge and SQS mock clients). Otherwise it relies on `Test/compile` plus
CI integration tests.
- Docker isn't available locally, so integration tests are recorded as "not
run locally" in `Tests`.
### Groups (in execution order)
Groups that don't touch files changed by open PRs go first. Google, MQTT and
AWS go last, so #2045/#2047/#2048/#2046 may have merged by then; otherwise I
rebase and resolve conflicts.
1. **AMQP** (`amqp`)
- Remove the default argument from `AmqpRpcFlow.committableFlow`; add
`createSimple(settings)`.
- `withSSLContext(SSLContext)` and `Optional` overloads.
- `Pair[String, Int]`→`Pair[String, java.lang.Integer]` (in place +
exclude).
- Java getters on `WriteMessage`, `*SourceSettings`, `AmqpWriteSettings`,
`*Declaration`, `AmqpDetailsConnectionProvider`.
2. **Cassandra** (`cassandra`)
- `withInit` ambiguity → new Java name + deprecate.
- `CassandraSession` constructors taking `Executor`.
- Java-friendly `CqlSessionProvider` adapter (`Executor` →
`CompletionStage`).
- Getters: `getMaxBatchWait`, `getInit`.
- Add parens to `serverMetaData` via an overload.
3. **Couchbase** (`couchbase`, `couchbase3`)
- couchbase:
- `@varargs` on `createIndex` (may need an exclude for the abstract
forwarder; check MiMa).
- `CompletionStage[java.lang.Boolean/Long]` (in place + exclude).
- Settings getters.
- couchbase3:
- `CouchbaseFlow` ambiguity → new Java names + deprecate.
- `exists` emits `java.lang.Boolean`.
- `CouchbaseSource` explicit overloads (matching
`CouchbaseFlow`/`CouchbaseSink`).
- `MutationDocument`/`MutationBinaryDocument` `create` factories and
`getResult(): Optional`.
4. **JMS** (`jms`, `jakartams`; identical code)
- `CustomDestination.create(name, java.util.function.Function)` and
deprecate the clashing constructor.
- `JmsObjectMessagePassThrough.withProperties(java.util.Map)`.
- `withHeaders(java.util.Set)`.
- `JmsMapMessagePassThrough.getBody`.
- Settings Java getters.
- `JmsTimeToLive.create(java.time.Duration)`.
5. **Hadoop / data** (`hbase`, `hdfs`, `kudu`)
- hbase: `withConverter` ambiguity → new Java name + deprecate; getters.
- hdfs: `RotationStrategy.time(java.time.Duration)`; remove the redundant
default argument from `HdfsSource.compressed`.
- kudu: `withConverter(java.util.function.Function)` and a getter.
6. **DB / search** (`elasticsearch`, `solr`, `mongodb`, `orientdb`,
`influxdb`, `slick`)
- elasticsearch:
- `getVersion` → `java.lang.Long` (in place + exclude).
- `WriteMessage` getters.
- `RetryLogic` Java base with `java.time.Duration`.
- `RetryNever.getInstance`.
- `ElasticsearchSink.create(MessageWriter)`.
- Mapper-less `ElasticsearchSource` overloads.
- Settings getters.
- solr:
- `WriteResult` and `WriteMessage` Java getters.
- `Operation` `getInstance`.
- mongodb: `replaceOne(collection)`; remove redundant default arguments.
- orientdb: `typed` overload without `query`; `withODatabasePool`.
- influxdb: `InfluxDbWriteResult.getError`;
`InfluxDbReadSettings.create()`; getters.
- slick:
- `nextByteArray(): Array[Byte]`, deprecating `nextBytes`.
- Javadsl `forDbAndProfile`.
- Javadsl `SlickWithTryResult` equivalent.
- Parameterised `source`.
- Deprecate the `ExecutionContextExecutor` overloads.
7. **I/O** (`file`, `ftp`, `unix-domain-socket`, `udp`, `sse`, `xml`, `text`)
- file:
- Javadsl `LogRotatorSink.withTypedSinkFactory`.
- `TarArchiveMetadata.getFilePathPrefix`.
- Wider `OpenOption` set and materialized types.
- Align the `Directory.walk` directory check.
- ftp:
- `FtpFile.getPermissions` and a `create` factory.
- Remove redundant default arguments.
- `Optional` getters.
- Declare `mkdirAsync` consistently.
- unix-domain-socket:
- `outgoingConnection(..., java.time.Duration)`, deprecating the Scala
`Duration` overload.
- `ServerBinding` unbind returning `CompletionStage[Done]`.
- Javadsl `bindAndHandle`.
- Short `outgoingConnection` overloads.
- udp: declare return types.
- sse: `create(..., retryDelay: java.time.Duration)`; overload without
`lastEventId`.
- xml:
- `getAttribute(name): Optional`.
- `getAttributesList`.
- Java `fromMapToAttributeList`.
- `Namespace.create(uri)`.
- `parserWithContext(Consumer)`.
- text: fix the `encoding` scaladoc.
8. **Misc** (`huawei-push-kit`, `azure-storage-queue`, `reference`,
`testkit`)
- huawei-push-kit:
- `Tokens.create(java.util.List)`.
- `java.util.List`/`Map` withers on `AndroidConfig`/`WebConfig`.
- `ConditionBuilder` `and`/`or`/`not`.
- `Optional` `getForwardProxy`; `ForwardProxy.create` with `Optional`.
- Fix the Java doc example that uses `Set.Set1`.
- azure-storage-queue: javadsl `fromFunction(Consumer)`.
- reference: typed `Resource.connection`; declare the
`attributes.resource` return type.
- testkit: `@java.lang.Deprecated` on `LogCapturingJunit4`.
9. **MQTT** (`mqtt`, `mqttv5`, `mqtt-streaming`)
- mqtt:
- `MqttMessage.getQos` and settings getters.
- `withOfflinePersistenceSettings` overloads.
- `MqttSubscriptions` getter.
- `MqttMessageWithAck.create`.
- mqttv5:
- `withAuth` ambiguity → new Java name with `java.lang.Integer`,
deprecating the old one.
- `withCleanStart` Scala-`Option` overload.
- Settings getters.
- `MqttMessageWithAck.create`.
- mqtt-streaming:
- `Event.getCarry`, `Publish.getPacketId`.
- `getInstance` on `PingReq`/`PingResp`/`Disconnect`.
- `Connect` Java withers and getters.
- `Subscribe`/`SubAck`/`Unsubscribe` Java list getters.
- Java getters on `DecodeError` and `MqttSessionSettings`.
- Document the int flags contract.
10. **AWS** (`s3`, `sqs`, `kinesis` incl. firehose, `dynamodb`)
- s3:
- `listObjectVersionsAndCommonPrefixes(Optional)`, deprecating the
`Option` overload.
- Explicit `request` overloads.
- `(…, system)` overloads for
`completeMultipartUpload`/`deleteUpload`/`putBucketVersioning`.
- `chunkSize`/`chunkingParallelism` overloads.
- `getObject` with `versionId` but no range.
- A `getObjectMetadata` overload.
- `BucketVersioningResult.getStatusOptional`. The existing `getStatus`
returns an `Option` and can't be overloaded by return type, so add a new name
and deprecate `getStatus`.
- `Optional[java.lang.Boolean]` (in place + exclude).
- `java.util.Map` withers on `S3Headers`/`MetaHeaders`.
- `S3Settings` Java `allowedHeaders` create, wither and getter.
- `RetrySettings.defaults()` and `create(Config)`.
- Model `create`, `withLocation` and getters.
- sqs:
- `withVisibilityTimeout(java.time.Duration)`.
- `getResponseMetadata` (deprecate the `geResponseMetadata` typo).
- Typed `getResult` overrides.
- Settings getters.
- `SqsPublishGroupedSettings.create(java.time.Duration)`.
- `getRequest`; naming aliases.
- kinesis:
- `SchedulerBuilder` Java factory taking a `Function`.
- `CommittableRecord.getShutdownReason`.
- Firehose `create` aliases.
- Javadsl `byPartitionAndData`/`byPartitionAndBytes`.
- Settings getters.
- dynamodb: `flowWithContext` variant with a Java result type (it is
`@ApiMayChange`).
11. **Google** (`google-common`, `google-cloud-bigquery`,
`google-cloud-bigquery-storage`, `google-cloud-pub-sub`,
`google-cloud-pub-sub-grpc`, `google-cloud-storage`, `google-fcm`)
- google-common:
- Make `Google` callable statically: rename the `private[connectors]`
trait so `object Google` gets static forwarders, with an exclude for the trait
rename.
- `getForwardProxy` returning `Optional` under a new name, deprecating
the old one.
- `Credentials.create`.
- Remove `@InternalApi` from `RequestSettings`.
- bigquery:
- `getLabels`/`getMaximumBytesBilled` Java types (in place + exclude).
- `Job` marshallers and type-parameter rename.
- `create` alias; `listDatasets()`.
- bigquery-storage:
- New `Pair`-returning methods replacing the `Tuple2` ones, which get
deprecated.
- Java-protobuf `TableReadOptions` overloads.
- `BigQueryRecord.getOptional`, `fromMap(java.util.Map)`.
- Javadsl reader attributes.
- Consistent `maxNumStreams` defaults.
- pub-sub:
-
`PubSubMessage`/`PublishMessage`/`PublishRequest`/`AcknowledgeRequest` Java
getters.
- `publish` overloads without `parallelism`.
- pub-sub-grpc:
- `withUseTls`, `getRootCa`.
- `AckDeadlineDistribution.create(java.time.Duration)`.
- storage:
- Fix the "Java" `StorageObject.create` types with a new overload,
deprecating the old one.
- Readable
`Owner`/`ProjectTeam`/`CustomerEncryption`/`ObjectAccessControls`.
- `Optional[java.lang.Boolean]` holds (in place + exclude).
- Java `withMetadata`/`withAcl`.
- Bucket-operation overloads; primitive `chunkSize`.
- fcm:
- `java.util.Map`/`List` withers.
- `getInstance` on priority and visibility values.
- `ConditionBuilder` `and`/`or`/`not`.
- Getters and a `create()` alias.
- The bigquery-storage `Tuple2`→`Pair` change needs new method names,
because the erased signature is identical but the element type changes at
runtime. I'll propose names in the PR.
Group 10 (s3) and group 11 are large. If either exceeds about 1,500 changed
lines, I'll split it into two PRs (e.g. s3 / rest of AWS) and say so.
### Per-PR workflow
1. Branch from `main`.
2. Write the Java ambiguity or compile test first where relevant, and
confirm it fails.
3. Implement, following the conventions above.
4. Run `sbt "<mod>/headerCreateAll"`. Run `scalafmt --mode
diff-ref=upstream/main` and scalafmt on new files. With JDK 17, run `sbt
"<mod>/javafmtAll"`.
5. Run the scoped checks:
```shell
sbt "<mod>/Test/compile" "<mod>/Test/testOnly <new Java/Scala tests>"
"<mod>/mimaReportBinaryIssues" …
```
This covers every touched module. Then run `git diff --check`, ignoring
CRLF docs files.
6. Update paradox docs where snippets or APIs changed; run `sbt
docs/paradox` if docs changed.
7. Open the PR with `Refs #2043`.
8. Tick the matching boxes in this issue once the PR is open.
### Verification
- **Each PR:** the new Java API tests pass, and `Test/compile` passes for
every touched module, which proves Java call sites compile without casts,
`$MODULE$` or Scala types. Scoped MiMa passes, with only the agreed
`IncompatibleSignatureProblem` excludes and explicitly justified ones.
- **Ambiguity fixes:** the Java test that failed to compile before the fix
compiles after it.
- **CI:** `Check / Binary Compatibility` and the integration-test jobs,
which can't run locally without Docker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
