This is an automated email from the ASF dual-hosted git repository.
martijnvisser pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/flink-connector-gcp-pubsub.git
from fabf8fa [hotfix] Add licensing
new 7b2a81e [FLINK-9311] [pubsub] Clean up / add documentation and style
issues in the PubSub connector
new 58ea0ec [FLINK-9311] [pubsub] Improvements to builders + minor
improvement to PubSubSink flush logic
new e665389 Update version to 1.10-SNAPSHOT
new d94f878 [FLINK-13231] [pubsub] Replace Max outstanding
acknowledgement ids limit with a FlinkConnectorRateLimiter
new 1af4576 [FLINK-14654][logging] Fix various placeholder issues
new fa543ee Update version to 1.11-SNAPSHOT
new adb4a5d [FLINK-15672][build] Migrate to log4j2
new 6f3bb77 [FLINK-15672][build][tests] Migrate test log4j configurations
to log4j2
new e2536e1 [FLINK-15090][tests] Explicitly depend on flink-clients
new f261be7 [FLINK-16572] [pubsub,e2e] Only acknowledge list of messages
if the list is not empty + small style fixes (removal of uncommented code etc)]
new 3cd4c23 [FLINK-17306] Add open to PubSubDeserializationSchema
new 5ea50cf [FLINK-17306] Call open of SerializationSchema in PubSub sink
new c16b5b9 [FLINK-17307] Add collector to deserialize in PubSub
new 6a29a3b Update version to 1.12-SNAPSHOT
new ed0d5cc [FLINK-8871][checkpoint] Support to cancel checkpoing via
notification on task side
new f89d069 Merge pull request #12301 from Xeli/FLINK-16572-logs
new d0e5fc8 [FLINK-16572] Clean up PubSub connector e2e test
new bf80cb0 [hotfix][pubsub] Use TestLogger
new 648dce9 [FLINK-16572][e2e][pubsub] Acknowledge message in previous
test
new a7eec56 [FLINK-18607][build] Give the maven module a human readable
name
new 29c47fd [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
new c1518b5 [FLINK-19619][e2e] Pin CloudSDK version to have access to
pubsub emulator in test
new 45d9196 [FLINK-18363] Add user classloader to context in
DeSerializationSchema
new 032d137 [FLINK-19698] Move the CheckpointListener from flink-runtime
to flink-core. To maintain backwards compatibility, the CheckpointListener in
flink-runtime is not deleted immediately.
new cfc4293 Update version to 1.13-SNAPSHOT
new c06e25a [FLINK-20651] Format code with Spotless/google-java-format
new 54d5905 [FLINK-21193][docs] Migrate Flink docs from Jekyll to Hugo
new 1155ebe [hotfix][docs] Correct rendering of maven modules
new 8b30977 Update version to 1.14-SNAPSHOT
new 4963e90 [FLINK-23084] Move Executors/-ThreadFactory to flink-core
new ce030a1 [hotfix][docs] Fix variable name
new 01df040 [FLINK-18783] Load Akka with separate classloader
new a817846 Revert "[FLINK-18783] Load Akka with separate classloader"
new 4e8e84b [FLINK-18783] Load Akka with separate classloader
new 23c51dd [FLINK-23652][core/metrics] Extract Operator(IO)MetricGroup
interfaces and expose them in RuntimeContext
new 9656f72 Update version to 1.15-SNAPSHOT
new 4e5babe [FLINK-24018][build] Remove Scala dependencies from Java APIs
new 1f9c82c [FLINK-25159][tests] Streamline E2E surefire setup
new a5d7dc8 [FLINK-25129][docs] Restructure project configuation pages
new 1671de8 [FLINK-25892][gcp-pubsub][test] add ArchUnit tests for test
code
new cd7cd49 [hotfix] Fix JUnit 5 mini cluster arch unit rules
new c3aae7e [FLINK-26294][test] Using fixed description for ArchUnit
ITCaseRules
new 1ada07b [FLINK-26159][doc] add description for MAX_FETCH_RECORD
related question
new e612b32 Update version to 1.16-SNAPSHOT
new 562b5c4 [FLINK-26578][docs-zh] Translate new Project Configuration
section to Chinese. This closes #19100
new 3e4c3fe [FLINK-27034][gcp][tests] Use testcontainers
new 28c809c [hotfix][docs-zh] Fix "Google Cloud PubSub" Chinese page
under "DataStream Connectors" (#19483)
new cdb0f38 [FLINK-24721][build] Remove unnecessary relativePath to parent
new dc7c662 [FLINK-27937][tests][pubsub] Migrate
flink-connectors-gcp-pubsub to JUnit5
new ed8610f [FLINK-24433][Tests][Buildsystem] Turn off debug logging for
all end-to-end tests to avoid flooding the disk space
new e860c75 [FLINK-30058][Connector/Google PubSub] Move existing Google
Cloud PubSub connector code from Flink repo to dedicated Google PubSub repo
new 55aa842 [FLINK-30058][Tests] Add DockerImageVersions
new bb241f7 [FLINK-30058][Architecture] Add ProductionCodeArchitectureTest
new 7a2e541 [FLINK-30058][Build] Add Flink-style IntelliJ settings
new c546198 [FLINK-30058][Connector/Google PubSub] Add release tools as
submodule
The 55 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.gitmodules | 4 +
.idea/vcs.xml | 25 ++
.../docs/connectors/datastream/pubsub.md | 153 ++++++++
docs/content/docs/connectors/datastream/pubsub.md | 153 ++++++++
flink-connector-gcp-pubsub-e2e-tests/pom.xml | 111 ++++++
.../gcp/pubsub/CheckPubSubEmulatorTest.java | 130 +++++++
.../gcp/pubsub/EmulatedFullTopologyTest.java | 222 ++++++++++++
.../gcp/pubsub/EmulatedPubSubSinkTest.java | 148 ++++++++
.../gcp/pubsub/EmulatedPubSubSourceTest.java | 154 ++++++++
.../SimpleStringSchemaWithStopMarkerDetection.java | 38 ++
.../gcp/pubsub/emulator/GCloudUnitTestBase.java | 82 +++++
.../gcp/pubsub/emulator/PubsubHelper.java | 225 ++++++++++++
.../gcp/pubsub/test/DockerImageVersions.java | 29 ++
.../src/test/resources/log4j2-test.properties | 21 +-
.../0e990728-73ca-4506-b7ad-bbcd6750ba22 | 0
.../2f5a722a-a4be-46cf-977f-047a2a23bf90 | 0
.../523c7827-2ffa-4081-bbb3-d09679b7bbb8 | 0
.../8bf89fe2-7522-424e-ac41-c271d0809c09 | 0
.../97f3780e-2fb4-43fd-a58c-09ea654509a8 | 0
.../b15f3843-3603-447d-b9ac-f63abafb89e3 | 0
.../d8268f41-1f32-4ad5-ba10-1e9114c420f9 | 0
.../fbd5656f-4de9-46c3-9569-4fe1c6260b59 | 0
.../archunit-violations/stored.rules | 10 +
flink-connector-gcp-pubsub/pom.xml | 104 ++++++
.../gcp/pubsub/BlockingGrpcPubSubSubscriber.java | 139 +++++++
.../gcp/pubsub/DefaultPubSubSubscriberFactory.java | 73 ++++
.../gcp/pubsub/DeserializationSchemaWrapper.java | 63 ++++
.../connectors/gcp/pubsub/PubSubSink.java | 362 +++++++++++++++++++
.../connectors/gcp/pubsub/PubSubSource.java | 398 ++++++++++++++++++++
.../pubsub/common/AcknowledgeIdsForCheckpoint.java | 53 +++
.../gcp/pubsub/common/AcknowledgeOnCheckpoint.java | 129 +++++++
.../connectors/gcp/pubsub/common/Acknowledger.java | 26 ++
.../pubsub/common/PubSubDeserializationSchema.java | 81 +++++
.../gcp/pubsub/common/PubSubSubscriber.java | 33 ++
.../gcp/pubsub/common/PubSubSubscriberFactory.java | 37 ++
.../gcp/pubsub/emulator/EmulatorCredentials.java | 62 ++++
.../emulator/EmulatorCredentialsProvider.java | 37 ++
.../PubSubSubscriberFactoryForEmulator.java | 76 ++++
.../ProductionCodeArchitectureTest.java | 41 +++
.../architecture/TestCodeArchitectureTest.java | 40 +++
.../pubsub/DeserializationSchemaWrapperTest.java | 93 +++++
.../connectors/gcp/pubsub/PubSubConsumingTest.java | 242 +++++++++++++
.../connectors/gcp/pubsub/PubSubSourceTest.java | 161 +++++++++
.../pubsub/common/AcknowledgeOnCheckpointTest.java | 146 ++++++++
.../src/test/resources/archunit.properties | 31 ++
.../src/test/resources/log4j2-test.properties | 22 +-
pom.xml | 400 +++++++++++++++++++++
tools/releasing/shared | 1 +
48 files changed, 4331 insertions(+), 24 deletions(-)
create mode 100644 .gitmodules
create mode 100644 .idea/vcs.xml
create mode 100644 docs/content.zh/docs/connectors/datastream/pubsub.md
create mode 100644 docs/content/docs/connectors/datastream/pubsub.md
create mode 100644 flink-connector-gcp-pubsub-e2e-tests/pom.xml
create mode 100644
flink-connector-gcp-pubsub-e2e-tests/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/CheckPubSubEmulatorTest.java
create mode 100644
flink-connector-gcp-pubsub-e2e-tests/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/EmulatedFullTopologyTest.java
create mode 100644
flink-connector-gcp-pubsub-e2e-tests/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/EmulatedPubSubSinkTest.java
create mode 100644
flink-connector-gcp-pubsub-e2e-tests/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/EmulatedPubSubSourceTest.java
create mode 100644
flink-connector-gcp-pubsub-e2e-tests/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/SimpleStringSchemaWithStopMarkerDetection.java
create mode 100644
flink-connector-gcp-pubsub-e2e-tests/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/emulator/GCloudUnitTestBase.java
create mode 100644
flink-connector-gcp-pubsub-e2e-tests/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/emulator/PubsubHelper.java
create mode 100644
flink-connector-gcp-pubsub-e2e-tests/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/test/DockerImageVersions.java
copy .github/workflows/push_pr.yml =>
flink-connector-gcp-pubsub-e2e-tests/src/test/resources/log4j2-test.properties
(70%)
create mode 100644
flink-connector-gcp-pubsub/archunit-violations/0e990728-73ca-4506-b7ad-bbcd6750ba22
create mode 100644
flink-connector-gcp-pubsub/archunit-violations/2f5a722a-a4be-46cf-977f-047a2a23bf90
create mode 100644
flink-connector-gcp-pubsub/archunit-violations/523c7827-2ffa-4081-bbb3-d09679b7bbb8
create mode 100644
flink-connector-gcp-pubsub/archunit-violations/8bf89fe2-7522-424e-ac41-c271d0809c09
create mode 100644
flink-connector-gcp-pubsub/archunit-violations/97f3780e-2fb4-43fd-a58c-09ea654509a8
create mode 100644
flink-connector-gcp-pubsub/archunit-violations/b15f3843-3603-447d-b9ac-f63abafb89e3
create mode 100644
flink-connector-gcp-pubsub/archunit-violations/d8268f41-1f32-4ad5-ba10-1e9114c420f9
create mode 100644
flink-connector-gcp-pubsub/archunit-violations/fbd5656f-4de9-46c3-9569-4fe1c6260b59
create mode 100644 flink-connector-gcp-pubsub/archunit-violations/stored.rules
create mode 100644 flink-connector-gcp-pubsub/pom.xml
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/BlockingGrpcPubSubSubscriber.java
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/DefaultPubSubSubscriberFactory.java
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/DeserializationSchemaWrapper.java
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubSink.java
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubSource.java
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/common/AcknowledgeIdsForCheckpoint.java
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/common/AcknowledgeOnCheckpoint.java
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/common/Acknowledger.java
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/common/PubSubDeserializationSchema.java
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/common/PubSubSubscriber.java
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/common/PubSubSubscriberFactory.java
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/emulator/EmulatorCredentials.java
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/emulator/EmulatorCredentialsProvider.java
create mode 100644
flink-connector-gcp-pubsub/src/main/java/org/apache/flink/streaming/connectors/gcp/pubsub/emulator/PubSubSubscriberFactoryForEmulator.java
create mode 100644
flink-connector-gcp-pubsub/src/test/java/org/apache/flink/architecture/ProductionCodeArchitectureTest.java
create mode 100644
flink-connector-gcp-pubsub/src/test/java/org/apache/flink/architecture/TestCodeArchitectureTest.java
create mode 100644
flink-connector-gcp-pubsub/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/DeserializationSchemaWrapperTest.java
create mode 100644
flink-connector-gcp-pubsub/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubConsumingTest.java
create mode 100644
flink-connector-gcp-pubsub/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/PubSubSourceTest.java
create mode 100644
flink-connector-gcp-pubsub/src/test/java/org/apache/flink/streaming/connectors/gcp/pubsub/common/AcknowledgeOnCheckpointTest.java
create mode 100644
flink-connector-gcp-pubsub/src/test/resources/archunit.properties
copy .github/workflows/push_pr.yml =>
flink-connector-gcp-pubsub/src/test/resources/log4j2-test.properties (70%)
create mode 100644 pom.xml
create mode 160000 tools/releasing/shared
