poorbarcode opened a new pull request, #127: URL: https://github.com/apache/flink-connector-pulsar/pull/127
## Purpose of the change Support Flink 2.0.x and Pulsar Client 4.0.9 We compile against Flink 2.0.x intentionally. The connector only uses stable Flink 2.x Source/Sink APIs and does not depend on any Flink 2.2-specific API. Flink 2.2 compatibility is covered by testing; a dedicated -2.2 artifact is unnecessary unless a future change requires a 2.2-only API. I propose upgrading the Pulsar client directly to 4.0.12, the latest maintenance release in the Pulsar 4.0 LTS line, rather than upgrading only to 3.0.17 as proposed in [PR #125](https://github.com/apache/flink-connector-pulsar/pull/125). Pulsar 4.0.12 includes accumulated bug fixes and stability improvements, and avoids releasing a new connector version with an older client dependency. This upgrade should also replace the internal PulsarClientImpl#getPartitionedTopicMetadata call with the public PulsarClient#getPartitionsForTopic API, assuming it provides the required equivalent behavior. In addition, [Apache Pulsar PR #26073](https://github.com/apache/pulsar/pull/26073) introduces an enhancement specifically for flink-connector-pulsar. Once this change is available in an official Pulsar release, I plan to leverage it in the connector to further improve its stability. ## Brief change log - Adapt the connector to the API changes introduced in Flink 2.0. - Copy the implementation of the removed DataTypeUtils.stripRowPrefix static method into the connector, as the project still depends on this functionality. - Use the new PulsarClient#getPartitionedTopicMetadata API provided by the newer Pulsar client, replacing the previous internal-client implementation. - Add startup logging for the Source’s initial position. This adds minimal log volume while providing essential context for troubleshooting. - Add targeted debug-level logging to improve diagnostics without affecting normal log verbosity. - Fix duplicate message delivery in PulsarPartitionSplitReader.fetch. Besides correcting the reader behavior, this change reduces the intermittent failures that can make CI less reliable. - Print additional diagnostic logs when tests time out, making CI failures easier to investigate. - Introduce a temporary SimpleCollectIteratorAssert to improve diagnostics after CI failures. Unlike the existing assertion helper, it does not try to consume every remaining message after a failure, since the server may no longer emit messages in the test-failing scenario. This prevents the diagnostic path itself from hanging while waiting for additional messages. - I will investigate the underlying delivery issue in a follow-up PR and remove this temporary helper once we confirmed it is unnecessary. - Change the test log output stream from SYSTEM_ERR to SYSTEM_OUT, since normal diagnostic logs should not be written to the error stream. ## Significant changes *(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)* - [x] Dependencies have been added or upgraded - [x] Public API has been changed (Public API is any class annotated with `@Public(Evolving)`) - [ ] Serializers have been changed - [ ] New feature has been introduced - If yes, how is this documented? (not applicable / docs / JavaDocs / not documented) ## Additional notes [PR #123](https://github.com/apache/flink-connector-pulsar/pull/123) is working on a similar set of changes, but it has not been updated for nearly two months. I am opening this PR to help move the work forward. If the author of PR #123 resumes work and updates it before this PR's CI has passed, I am happy to close this PR and continue the discussion there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
