david-streamlio opened a new issue, #99:
URL: https://github.com/apache/pulsar-connectors/issues/99
## Description
The Canal source connector's client library
(`com.alibaba.otter:canal.client:1.1.7` / `canal.protocol:1.1.7`) is
incompatible with the protobuf-java version the build resolves. Its generated
`CanalPacket` code calls
`com.google.protobuf.GeneratedMessageV3.makeExtensionsImmutable()`, a method
**removed** in modern protobuf-java (the 3.21+/4.x line). So
`CanalConnector.connect()` throws `NoSuchMethodError` and the source can never
complete the canal-server handshake.
Found while adding the first integration test for the module (#48 / #98) —
the module had zero tests, so this was never exercised.
## Evidence
On the canal module's **test** classpath, protobuf-java resolves to 4.31.1
(pulled by `pulsar-client`/`pulsar-functions-instance`), and
`GeneratedMessageV3` in that version has no `makeExtensionsImmutable`. The
integration test reproduces the `NoSuchMethodError` on `connect()` without a
version pin.
## Nuance worth a maintainer's eye — is production affected?
- The module's **runtimeClasspath** (what the NAR bundles) resolves **no**
external `protobuf-java` at all, and `canal.protocol-1.1.7.jar` does not bundle
protobuf classes. So the NAR appears to rely on whatever protobuf the Pulsar
runtime provides — which is modern — meaning the connector would very likely
hit the same `NoSuchMethodError` **in production**, not just in the test. This
should be confirmed against a real deployment.
## The fix in #98, and why it needs review
#98 makes the integration test pass by forcing `protobuf-java` to **3.6.1**
(canal's declared version) via `configurations.all { resolutionStrategy {
force(...) } }`. That works, but:
1. **3.6.1 is from 2018** and carries known CVEs (e.g. CVE-2021-22569,
CVE-2022-3171). Pinning the whole module — including the shipped NAR — to it is
heavy.
2. The demonstrated breakage is on the **test** classpath; whether the NAR
needs the pin depends on the production question above. If production is
unaffected, the force should be scoped to test configurations only.
3. **`canal.client` 1.1.8 exists** (latest). If it is built against a newer
protobuf, upgrading is a cleaner fix than downgrading protobuf. Worth checking
before committing to the 3.6.1 pin.
## Suggested resolution
Decide among: (a) upgrade to `canal.client`/`canal.protocol` 1.1.8 if
protobuf-4-compatible; (b) shade/relocate protobuf for the canal NAR; or (c)
keep the 3.6.1 pin but scope it correctly (test-only if production is
unaffected) and accept the CVE tradeoff. #98 currently takes (c) module-wide;
flagging for a decision.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
