This is an automated email from the ASF dual-hosted git repository. sijie pushed a change to branch branch-2.1 in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git.
omit d3261ed Enable authentication and authorization for some admin APIs (#2046) omit fde1da6 Fix license in binary distribution (#2040) omit f66b74b Fixed resetting of producer sequence id counter after receiving SendError before any message was successfully published (#2047) omit 94484a5 Fixed lookups for v2 topics in C++ client lib with HTTP service URL (#2043) omit 1d32b75 Add depedency on grpcio and protobuf for Python lib (#2044) omit de7332b Move AuthenticationFilter to pulsar-broker-common (#2039) omit 30f6128 Exclude a few files from apache-rat:check (#2035) omit 3fbe4ac Added custom compiled OpenSSL into docker image for RPM/Deb omit e959f09 Added integer types casting to avoid compile errors with gcc-4.8 (#2031) omit 7f8985a Fixed C99 variable initialization in cgo header (#2029) omit 840ad88 Update LICENSE & NOTICE for newly introduced or version-upgraded dependencies (#2023) omit 10397f7 Remove jboss-reflect dependency (#2022) omit 270dcf1 Release 2.1.0-incubating add 6eb238b Remove jboss-reflect dependency (#2022) add 7a5072e Update LICENSE & NOTICE for newly introduced or version-upgraded dependencies (#2023) add 8a8bef7 Fixup misleading text in compaction cookbook (#2026) add 172f050 Added message parser to decode raw headers and payload without a Consumer instance (#2019) add f7c5696 adding pulsar io docs (#2001) add b506793 Helm charts for deployment on GKE (#1993) add 8b2929b Bump Pulsar version to 2.2.0-incubating (#2027) add 97d06c9 Fixed C99 variable initialization in cgo header (#2029) add ae90f22 Change MessageProcessor Interface to be public (#2030) add bdf52bc Added integer types casting to avoid compile errors with gcc-4.8 (#2031) add ca93d4a Added custom compiled OpenSSL into docker image for RPM/Deb (#2032) add b0292a1 adding a module that shades managed-ledge (#2034) add 174fd02 Exclude a few files from apache-rat:check (#2035) add c80dfb6 Move AuthenticationFilter to pulsar-broker-common (#2039) add 7c30702 Fix license in binary distribution (#2040) add 4487d04 Add depedency on grpcio and protobuf for Python lib (#2044) add ae596ed Inject AuthenticationService into ProxyService (#2038) add c16dfbe Original principal authorization for admin API (#2028) add 398a0bd Enable authentication and authorization for some admin APIs (#2046) add 583ddee Fixed lookups for v2 topics in C++ client lib with HTTP service URL (#2043) add f1df048 Fixed resetting of producer sequence id counter after receiving SendError before any message was successfully published (#2047) add 99c0ed2 Issue 1416: Docs about garbage collection are inaccurate/lacking (#2049) add 4d1cf20 Introduce testcontainer based cluster integration tests (#2055) add bee124e Add semantic integration tests for non-persistent topics and effectively-once producing (#2059) add f14b9e9 fix bug with specifying jar via config file (#2053) add 7cebe23 Forward user-properties to sink (#2057) add 89c7b65 Pulsar Kinesis-sink upgrade aws-sdk version (#2058) add 9b1cc78 Forward encryption properties with encrypted payload to consumer (#2024) add 60a07e9 Migrate CLI integration test mode to use testcontainers (#2060) add 124dfaf Move integration tests for function upload/download to apache (#2061) add 5766844 fix bug in concerning ContextImpl (#2052) add 4ade1bd Tests Python Wheel packaging in CI build (#2042) add b22e883 Package Pulsar IO connectors into separate tgz in form of nar archives (#2054) add d01cc75 Fix build issue in master (#2062) add 791b596 Add integration tests on pulsar functions on process mode (#2063) add 7a10d93 GCS offload support(1): rename s3offload related classes to be reuse-able (#2064) add 7559632 Improve RedeliverUnacknowledgedMessages binary protocol docs (#2066) add e16f91c Run functions runtime test with thread mode (#2070) add 5e4579d Pass encryption-context to pulsar-sink if source receive encrypted message (#2068) add 5614cc5 Integration test coverage on python functions (#2072) add 5d73bb4 Fix: destination lookup url for v2 (#2075) add 8e95ddc By default PulsarSource consume encrypted message (#2074) add e422a6f fix: configure crypto-action for internal consumer independently cryptoReader (#2073) add 9a8428a shade dependencies in pulsar client and admin (#2069) add 59fc8e2 Dump container & pulsar log before container stops (#2076) add c9d1066 Fix python/CMakeLists.txt (#2084) add ec04dc0 Move Pulsar IO as a top-category section at the sidebar (#2050) add 1893323 change JSONSchema to generate and store an avro schema (#2071) add d62cdd8 Cleanup unused function resource (#2082) add f71f4ba avoid unnecessary reflection to create pulsar source instance (#2083) add 4eee5b1 Added ReadOnlyCursor to ManagedLedger (#2041) add 2f6dc80 Fix Rackaware test (#2094) add 99892d7 Enhance Pulsar cluster with container logs (#2078) add 268161c Move Smoke Test and Cli test to semantic package (#2097) add 0420e59 Fix wget argument (#2085) add 83e3157 Issue 1455: MessageID has always batch index 0 when sending messages in a batch (#2099) add 3e9bac6 Issue 1252: Enable bookie autorecovery daemon by default (#2096) add da8981c Kinesis sink publish full json message (#2079) add 7bfc343 Remove dependencies exclusions for org.apache.httpcomponents:httpclient (#2104) add 1ebe334 Fixed path for Postgres in dashboard docker image (#2088) add 86f0a0d Remove unused test annotation (#2095) add 423a9af Add context to source and sink (#2098) add ffe7b62 Issue 2077: SubscriptionInitialPosition doesn't work with multiple-topic type subscription (#2100) add 027f535 Issue 2090: (#2092) add 5402d21 Move Security related documentation to a top-level category (#2051) add 2bf2568 Load connectors from NAR archives (#2102) add c455e5e Provide API and CLI to access list of connectors supported by system (#2106) add bc6e1e6 Disable empty methods in SequenceIdWithErrorTest (#2093) add d81cd38 Make ack timeout work in asynchronous implementation of C++ consumer (#2087) add 7047146 derive worker-host and id at runtime if not provided (#2113) add cd35677 change ID() to Payload (#2117) add b113e90 Make it possible to set idle WebSocket session timeout period (#2118) add 8b206c3 patching CmdSink and adding extensive unit tests (#2107) add a92ac3d add { in creation of ClientOptions (#2115) add 5c9d077 shading dependencies in pulsar client (#2120) add 914de19 Only exit from watch-znode if we managed to communicate with zk (#2127) add a89fe84 remove | (#2125) add db804e7 Remove popover from security overview page (#2109) add 5632d95 Fix an Argument of Wget in Pulsar Functions Getting Started (#2132) add 6391b84 patching CmdSources and adding unit tests (#2124) add 76fb8dc Fix a few links in security pages (#2134) add 133331c Extract out common code from standalone and build Embedded Pulsar (#2080) add 2147201 Submit and run locally builtin connectors (#2114) add a4e9399 Add pulsar python to gitignore (#2123) add 8631067 Temporarely disable SequenceIdWithErrorTest (#2140) add 15e4d3d fix link in binary protocol (#2131) add bf906d1 Fix a command in Pulsar Function overview Document (#2143) add cc8ec3d Issue 1433: Expose batch flushAsync() and flush() methods in Producer (#2103) add 4843154 Fixed mem leak when receiving unrecoverable fencing errors in managed ledger (#2138) add 960dc3a Fixed RPM/Deb package to include "apache" and "incubating" in name (#2129) add 387f884 Added more Java and Python examples and made the examples match across languages (#2137) add ede2108 Release resources for ZLib inflater/deflater when not used (#2111) add 2f38b89 Minor bug-fixes to examples (#2145) add 14fc5d9 Schedule task to update function stats separately (#2128) add d43bf87 add list-connector rest api at workers (#2146) add 2f47a10 Slow consumer prevents offloaded ledger cleanup (#2126) add 8736582 Add authentication to admin proxy (#2056) add e071ec1 Use container name in logs (#2139) add 2a6d195 add flatbuffer option to serialize kinesis-message in KinesisSink (#2108) add e9d67bc Function: REST and CLI to get list of workers in cluster (#2112) add 1e2b5d7 Add std=c99 to CMAKE_C_FLAGS to avoid compile error with gcc-4.8 (#2086) add 06e26b2 Fix schema initialization (#2148) add 123524d Avoid closing cursor when consumers are disconnected (#2149) add 26a70db Fix: function load correct type-class of connector (#2147) add 3b8faea Fix: file-url sink/source submission with cli (#2150) add ea6eae7 Added optional key in pulsar IO (#2116) add 10f273a Add numFunctionWorkers and externalServices to cluster spec (#2160) add e54c58a fix: conflicting jackson transitive dependency for worker (#2159) add 0ff99e7 Fix: trigger function rest-api (#2162) add ce9b125 git ignore nar extension file (#2161) add 35a121b Add a new docker image to include all pulsar io connectors (#2158) add 54dcee4 bin/bookkeeper doesn't output the result to the console (#2155) add 300f736 Documentation need to be updated for Admin command to apply deduplication at namespace: (#2157) add b2ef2e4 Add pulsar-version comment for future reference (#2163) add 555dd57 FunctionActioner should set the type class name for both source and sink (#2167) add 7af6374 fixing protobuf schema shading issue (#2168) add 5f779b4 Add function metrics with function-stats to get metrics on-demand (#2130) add ba1ea66 Add kinesis-sink user metrics to sinkContext (#2169) add da5b3cd If auth is badly configured in c++, close the connection (#2172) add 5b72eec Issue 2119: TopicPatternSubscription doesn't work through proxy (#2176) add 7b59248 Migrate compaction, s3offload to test containers (#2101) add d5536cc Removed GPL deps on gnu crypto from aerospike connector (#2173) add 918e38c Inherit I/O streams in functions process runtime (#2178) add d6fb900 Add integration test for kafka sink (#2170) add 8de127d Add integration test for kafka source (#2171) add 3dfc217 Don't dirty the tree when building in CI (#2175) add f0024d2 Issue 2121: Improve logging around "Namespace not served by this instance" (#2183) add fd7b32c Issue 1288: Provide documentation for running BookKeeper auto-recovery (#2186) add b3845b2 Moved Record interface as part of functions api (#2184) add 6ef9df9 augmenting protoschema with info for parsing (#2181) add a5c339b Allow construction of c++ builtin auth plugins via factory (#2177) add d8c0f10 V2 doc changes (#2174) add e28ef1b Add support to configure subscription name for sink-function (#2198) add ffe3849 Improved dashboard documentation with warning on advertised address (#2196) add 266834e unshade avro in pulsar-client-schema (#2197) add dc06215 attach shaded artifact while deploying to maven-repo (#2202) add 98eeaf5 function-instance lookup: retrieve function instance owner's workerId (#2201) new a3082c1 Release 2.1.0-incubating This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (d3261ed) \ N -- N -- N refs/heads/branch-2.1 (a3082c1) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .gitignore | 6 +- .travis.yml | 2 +- bin/bookkeeper | 8 +- bin/pulsar | 4 +- bin/pulsar-admin | 10 +- bin/pulsar-client | 2 +- bin/pulsar-perf | 2 +- conf/bookkeeper.conf | 6 +- conf/broker.conf | 2 + conf/functions_worker.yml | 6 +- conf/log4j2.yaml | 24 +- conf/standalone.conf | 2 + conf/websocket.conf | 3 + dashboard/Dockerfile | 2 +- dashboard/conf/postgresql.conf | 10 +- dashboard/conf/supervisor-app.conf | 2 +- .../kubernetes/helm/README.md | 13 +- deployment/kubernetes/helm/pulsar/.helmignore | 21 + .../kubernetes/helm/pulsar/Chart.yaml | 8 +- .../kubernetes/helm/pulsar/templates/_helpers.tpl | 32 + .../pulsar/templates/autorecovery-configmap.yaml | 38 + .../pulsar/templates/autorecovery-deployment.yaml | 105 +++ .../helm/pulsar/templates/bastion-configmap.yaml | 27 +- .../helm/pulsar/templates/bastion-deployment.yaml | 80 ++ .../pulsar/templates/bookkeeper-configmap.yaml | 36 + .../helm/pulsar/templates/bookkeeper-pdb.yaml | 50 +- .../helm/pulsar/templates/bookkeeper-service.yaml | 51 +- .../pulsar/templates/bookkeeper-statefulset.yaml | 159 ++++ .../pulsar/templates/bookkeeper-storageclass.yaml | 59 ++ .../helm/pulsar/templates/broker-configmap.yaml | 40 + .../helm/pulsar/templates/broker-deployment.yaml | 114 +++ .../helm/pulsar/templates/broker-pdb.yaml | 50 +- .../helm/pulsar/templates/broker-service.yaml | 51 +- .../pulsar/templates/dashboard-deployment.yaml | 73 ++ .../helm/pulsar/templates/dashboard-service.yaml | 53 +- .../helm/pulsar/templates/grafana-deployment.yaml | 72 ++ .../helm/pulsar/templates/grafana-service.yaml | 53 +- .../helm/pulsar/templates/namespace.yaml | 9 +- .../pulsar/templates/prometheus-configmap.yaml | 70 ++ .../pulsar/templates/prometheus-deployment.yaml | 82 ++ .../helm/pulsar/templates/prometheus-pvc.yaml | 29 +- .../helm/pulsar/templates/prometheus-rbac.yaml | 57 ++ .../helm/pulsar/templates/prometheus-service.yaml | 53 +- .../pulsar/templates/prometheus-storageclass.yaml | 41 + .../helm/pulsar/templates/proxy-configmap.yaml | 42 + .../helm/pulsar/templates/proxy-deployment.yaml | 109 +++ .../helm/pulsar/templates/proxy-pdb.yaml | 52 +- .../helm/pulsar/templates/proxy-service.yaml | 53 +- .../helm/pulsar/templates/zookeeper-configmap.yaml | 25 +- .../helm/pulsar/templates/zookeeper-metadata.yaml | 58 ++ .../helm/pulsar/templates/zookeeper-pdb.yaml | 50 +- .../helm/pulsar/templates/zookeeper-service.yaml | 51 +- .../pulsar/templates/zookeeper-statefulset.yaml | 142 +++ .../pulsar/templates/zookeeper-storageclass.yaml | 39 + deployment/kubernetes/helm/pulsar/values.yaml | 405 +++++++++ .../terraform-ansible/templates/bookkeeper.conf | 2 +- {pulsar-io/kafka => distribution/io}/pom.xml | 57 +- distribution/io/src/assemble/README | 10 + distribution/io/src/assemble/io.xml | 84 ++ {pulsar-io/core => distribution}/pom.xml | 22 +- .../server}/licenses/LICENSE-AspectJ.txt | 0 .../server}/licenses/LICENSE-CC0.txt | 0 .../server}/licenses/LICENSE-CDDL-1.1.txt | 0 .../server}/licenses/LICENSE-EA-Agent-Loader.txt | 0 .../server}/licenses/LICENSE-EPL-1.0.txt | 0 .../server}/licenses/LICENSE-Hamcrest.txt | 0 .../server}/licenses/LICENSE-HdrHistogram.txt | 0 .../server}/licenses/LICENSE-HdrHistogram0.txt | 0 .../server}/licenses/LICENSE-JSR305.txt | 0 .../server}/licenses/LICENSE-LevelDB.txt | 0 .../server}/licenses/LICENSE-Lombok.txt | 0 .../server}/licenses/LICENSE-SLF4J.txt | 0 .../server}/licenses/LICENSE-SemVer.txt | 0 .../server}/licenses/LICENSE-Webbit.txt | 0 .../server}/licenses/LICENSE-bouncycastle.txt | 0 .../licenses/LICENSE-google-auth-library.txt | 0 .../server}/licenses/LICENSE-jbzip2.txt | 0 .../server}/licenses/LICENSE-jfastlz.txt | 0 .../server}/licenses/LICENSE-libdivsufsort.txt | 0 .../server}/licenses/LICENSE-protobuf.txt | 0 .../server}/licenses/LICENSE-xz.txt | 0 {all => distribution/server}/pom.xml | 6 +- .../server}/src/assemble/LICENSE.bin.txt | 2 + .../server}/src/assemble/NOTICE.bin.txt | 0 .../server}/src/assemble/README.bin.txt | 0 {all => distribution/server}/src/assemble/bin.xml | 20 +- {all => distribution/server}/src/assemble/src.xml | 2 +- docker/pom.xml | 1 + .../pulsar-all/Dockerfile | 9 +- docker/{pulsar => pulsar-all}/pom.xml | 75 +- docker/pulsar/Dockerfile | 3 +- docker/pulsar/pom.xml | 6 +- docker/pulsar/scripts/gen-yml-from-env.py | 81 ++ docker/pulsar/scripts/watch-znode.py | 40 +- managed-ledger-shaded/pom.xml | 291 ++++++ managed-ledger/pom.xml | 2 +- .../apache/bookkeeper/mledger/AsyncCallbacks.java | 6 + .../bookkeeper/mledger/ManagedLedgerFactory.java | 26 + .../apache/bookkeeper/mledger/ReadOnlyCursor.java | 101 +++ .../bookkeeper/mledger/impl/ManagedCursorImpl.java | 4 +- .../mledger/impl/ManagedLedgerFactoryImpl.java | 63 ++ .../bookkeeper/mledger/impl/ManagedLedgerImpl.java | 59 +- .../apache/bookkeeper/mledger/impl/OpAddEntry.java | 1 + .../mledger/impl/ReadOnlyCursorImpl.java | 65 ++ .../mledger/impl/ReadOnlyManagedLedgerImpl.java | 130 +++ .../mledger/impl/OffloadLedgerDeleteTest.java | 55 ++ .../mledger/impl/ReadOnlyCursorTest.java | 223 +++++ pom.xml | 85 +- pulsar-broker-auth-athenz/pom.xml | 2 +- .../apache/pulsar/broker/ServiceConfiguration.java | 10 + pulsar-broker/pom.xml | 9 +- ...tandaloneStarter.java => PulsarStandalone.java} | 322 ++++--- .../org/apache/pulsar/PulsarStandaloneBuilder.java | 124 +++ .../org/apache/pulsar/PulsarStandaloneStarter.java | 190 +--- .../org/apache/pulsar/broker/PulsarService.java | 2 +- .../pulsar/broker/admin/impl/FunctionsBase.java | 39 +- .../broker/admin/impl/PersistentTopicsBase.java | 4 +- .../apache/pulsar/broker/admin/v2/Functions.java | 7 + .../BackedInputStream.java} | 4 +- .../BlockAwareSegmentInputStream.java | 3 +- .../{s3offload => offload}/DataBlockHeader.java | 3 +- .../{s3offload => offload}/OffloadIndexBlock.java | 2 +- .../OffloadIndexBlockBuilder.java | 4 +- .../{s3offload => offload}/OffloadIndexEntry.java | 2 +- .../impl/BlockAwareSegmentInputStreamImpl.java | 4 +- .../impl/DataBlockHeaderImpl.java | 4 +- .../impl/OffloadIndexBlockBuilderImpl.java | 6 +- .../impl/OffloadIndexBlockImpl.java | 7 +- .../impl/OffloadIndexEntryImpl.java | 4 +- .../impl/S3BackedInputStreamImpl.java | 8 +- .../impl/S3BackedReadHandleImpl.java | 19 +- .../impl}/S3ManagedLedgerOffloader.java | 8 +- .../pulsar/broker/service/BrokerService.java | 4 +- .../apache/pulsar/broker/service/ServerCnx.java | 2 +- .../nonpersistent/NonPersistentReplicator.java | 7 +- .../service/persistent/PersistentReplicator.java | 7 +- .../service/persistent/PersistentSubscription.java | 28 +- .../broker/service/persistent/PersistentTopic.java | 22 +- .../service/schema/BookkeeperSchemaStorage.java | 114 ++- .../schema/JsonSchemaCompatibilityCheck.java | 117 ++- .../service/schema/SchemaCompatibilityCheck.java | 7 + .../pulsar/broker/web/PulsarWebResource.java | 65 +- .../pulsar/broker/admin/AdminApiTlsAuthTest.java | 322 +++++++ .../apache/pulsar/broker/admin/NamespacesTest.java | 55 +- .../S3BackedInputStreamTest.java | 18 +- .../broker/{s3offload => offload}/S3Mock.java | 2 +- .../broker/{s3offload => offload}/S3TestBase.java | 7 +- .../impl/BlockAwareSegmentInputStreamTest.java | 6 +- .../impl/DataBlockHeaderTest.java | 4 +- .../impl/OffloadIndexTest.java | 8 +- .../impl}/S3ManagedLedgerOffloaderTest.java | 12 +- .../pulsar/broker/service/RackAwareTest.java | 3 +- .../schema/AvroSchemaCompatibilityCheckTest.java | 127 +-- ...t.java => BaseAvroSchemaCompatibilityTest.java} | 69 +- .../schema/JsonSchemaCompatibilityCheckTest.java | 140 +++ .../schema/PartitionedTopicsSchemaTest.java} | 77 +- .../ProtobufSchemaCompatibilityCheckTest.java | 18 +- .../broker/service/schema/SchemaServiceTest.java | 5 + .../client/api/SimpleProducerConsumerTest.java | 234 ++++- .../api/SimpleTypedProducerConsumerTest.java | 8 +- .../pulsar/client/impl/MessageParserTest.java | 155 ++++ .../client/impl/SequenceIdWithErrorTest.java | 16 +- ...nkE2ETest.java => PulsarFunctionAdminTest.java} | 131 +-- .../org/apache/pulsar/io/PulsarSinkE2ETest.java | 128 ++- .../authentication/tls-http}/admin.cert.pem | 0 .../authentication/tls-http}/admin.key-pk8.pem | 0 .../authentication/tls-http}/broker.cert.pem | 0 .../authentication/tls-http}/broker.key-pk8.pem | 0 .../resources/authentication/tls-http}/ca.cert.pem | 0 .../authentication/tls-http/proxy.cert.pem | 26 + .../authentication/tls-http/proxy.key-pk8.pem | 28 + .../authentication/tls-http/superproxy.cert.pem | 26 + .../authentication/tls-http/superproxy.key-pk8.pem | 28 + pulsar-client-admin-shaded/pom.xml | 40 +- .../org/apache/pulsar/client/admin/Functions.java | 47 +- .../client/admin/internal/FunctionsImpl.java | 54 +- pulsar-client-cpp/docker/build-wheels.sh | 28 + pulsar-client-cpp/examples/CMakeLists.txt | 2 + pulsar-client-cpp/include/pulsar/Authentication.h | 8 +- pulsar-client-cpp/lib/Authentication.cc | 74 +- pulsar-client-cpp/lib/ClientConnection.cc | 10 + pulsar-client-cpp/lib/ConsumerImpl.cc | 1 + pulsar-client-cpp/lib/auth/AuthAthenz.h | 3 + pulsar-client-cpp/lib/auth/AuthTls.cc | 4 + pulsar-client-cpp/lib/auth/AuthTls.h | 3 + pulsar-client-cpp/pkg/deb/build-deb.sh | 19 +- pulsar-client-cpp/pkg/rpm/SPECS/pulsar-client.spec | 7 +- pulsar-client-cpp/pkg/rpm/build-rpm.sh | 2 +- pulsar-client-cpp/python/CMakeLists.txt | 5 +- .../python/pulsar/functions/context.py | 9 +- pulsar-client-cpp/python/pulsar_test.py | 84 +- pulsar-client-cpp/run-unit-tests.sh | 24 +- pulsar-client-cpp/tests/BasicEndToEndTest.cc | 81 ++ .../pulsar-client-kafka-shaded/pom.xml | 42 +- .../pulsar-client-kafka-tests/pom.xml | 2 +- .../pulsar-client-kafka/pom.xml | 2 +- pulsar-client-schema/pom.xml | 361 ++++++++ .../java/org/apache/pulsar/client/api/Schema.java | 16 + .../client/api/SchemaSerializationException.java | 0 .../pulsar/client/impl/schema/AvroSchema.java | 2 +- .../pulsar/client/impl/schema/BytesSchema.java | 0 .../pulsar/client/impl/schema/JSONSchema.java | 66 +- .../pulsar/client/impl/schema/ProtobufSchema.java | 135 +++ .../pulsar/client/impl/schema/StringSchema.java | 0 .../pulsar/client/schema}/AvroSchemaTest.java | 4 +- .../pulsar/client/schema/JSONSchemaTest.java | 27 +- .../pulsar/client/schema}/ProtobufSchemaTest.java | 55 +- .../src/test/proto/Test.proto | 5 +- pulsar-client-shaded/pom.xml | 45 +- pulsar-client-tools-test/pom.xml | 2 +- .../apache/pulsar/admin/cli/CmdFunctionsTest.java | 57 +- pulsar-client-tools/pom.xml | 44 + .../org/apache/pulsar/admin/cli/CmdFunctions.java | 84 +- .../java/org/apache/pulsar/admin/cli/CmdSinks.java | 337 ++++--- .../org/apache/pulsar/admin/cli/CmdSources.java | 282 ++++-- .../org/apache/pulsar/admin/cli/TestCmdSinks.java | 995 +++++++++++++++++++++ .../apache/pulsar/admin/cli/TestCmdSources.java | 858 ++++++++++++++++++ pulsar-client/pom.xml | 63 +- .../client/api/ConsumerCryptoFailureAction.java | 19 +- .../java/org/apache/pulsar/client/api/Message.java | 12 + .../org/apache/pulsar/client/api/Producer.java | 18 + .../pulsar/client/impl/BatchMessageContainer.java | 2 +- .../apache/pulsar/client/impl/ConsumerImpl.java | 65 +- .../pulsar/client/impl/HttpLookupService.java | 2 +- .../apache/pulsar/client/impl/MessageCrypto.java | 2 +- .../org/apache/pulsar/client/impl/MessageImpl.java | 36 +- .../apache/pulsar/client/impl/MessageParser.java | 191 ++++ .../pulsar/client/impl/MessageRecordImpl.java | 65 -- .../client/impl/MultiTopicsConsumerImpl.java | 21 +- .../client/impl/PartitionedProducerImpl.java | 12 +- .../apache/pulsar/client/impl/ProducerBase.java | 23 +- .../apache/pulsar/client/impl/ProducerImpl.java | 57 +- .../pulsar/client/impl/PulsarClientImpl.java | 3 +- .../apache/pulsar/client/impl/SendCallback.java | 14 +- .../pulsar/client/impl/TopicMessageImpl.java | 12 +- .../pulsar/client/impl/schema/ProtobufSchema.java | 82 -- .../pulsar/client/impl/MessageRecordImplTest.java | 60 -- .../{schemas => schema}/DefaultSchemasTest.java | 16 +- pulsar-common/pom.xml | 5 + .../org/apache/pulsar/common/api/Commands.java | 3 + .../pulsar/common/api/EncryptionContext.java | 52 +- .../apache/pulsar/common/api/proto/PulsarApi.java | 3 + .../common/compression/CompressionCodecZLib.java | 10 + .../pulsar/common/io/ConnectorDefinition.java | 35 +- .../org/apache/pulsar/common/nar/FileUtils.java | 221 +++++ .../apache/pulsar/common/nar/NarClassLoader.java | 280 ++++++ .../org/apache/pulsar/common/nar/NarUnpacker.java | 162 ++++ .../pulsar/common/util/ObjectMapperFactory.java | 28 +- pulsar-common/src/main/proto/PulsarApi.proto | 1 + pulsar-discovery-service/pom.xml | 4 +- .../org/apache/pulsar/functions/api/Record.java | 48 +- .../pulsar/functions/instance/ContextImpl.java | 39 +- .../pulsar/functions/instance/FunctionStats.java | 25 +- .../pulsar/functions/instance/JavaInstance.java | 28 +- .../functions/instance/JavaInstanceRunnable.java | 147 ++- .../pulsar/functions/instance/SinkRecord.java | 75 ++ .../apache/pulsar/functions/sink/PulsarSink.java | 72 +- .../pulsar/functions/source/PulsarRecord.java | 63 +- .../pulsar/functions/source/PulsarSource.java | 33 +- .../source/RecordWithEncryptionContext.java | 21 +- .../src/main/python/InstanceCommunication_pb2.py | 64 +- .../main/python/InstanceCommunication_pb2_grpc.py | 34 + .../instance/src/main/python/contextimpl.py | 28 +- .../instance/src/main/python/python_instance.py | 47 +- .../instance/src/main/python/server.py | 9 + .../functions/instance/JavaInstanceTest.java | 26 +- .../pulsar/functions/source/PulsarSourceTest.java | 10 +- ...unction.java => ConfigBasedAppendFunction.java} | 20 +- ...nterFunction.java => CustomObjectFunction.java} | 13 +- .../api/examples/ExclamationFunction.java | 4 + .../functions/api/examples/LoggingFunction.java | 18 +- .../functions/api/examples/PublishFunction.java | 8 +- .../functions/api/examples/UserMetricFunction.java | 6 +- .../functions/api/examples/VoidFunction.java | 3 + .../functions/api/examples/WindowFunction.java | 13 +- ...CounterFunction.java => WordCountFunction.java} | 9 +- .../api/examples/serde/CustomObject.java} | 21 +- .../api/examples/serde/CustomObjectSerde.java | 26 +- .../example-stateful-function-config.yaml | 2 +- .../pulsar/functions/worker/WorkerInfo.java} | 47 +- .../proto/src/main/proto/Function.proto | 9 + .../src/main/proto/InstanceCommunication.proto | 5 + ...function.py => config_based_append_function.py} | 10 +- .../{exclamation.py => custom_object_function.py} | 29 +- .../{logfunction.py => exclamation_function.py} | 5 +- .../{exclamation.py => logging_function.py} | 7 +- ...clamation.py => native_exclamation_function.py} | 0 .../{logfunction.py => publish_function.py} | 11 +- .../{exclamation.py => void_function.py} | 5 +- pulsar-functions/runtime-all/pom.xml | 1 + .../pulsar/functions/runtime/JavaInstanceMain.java | 30 + .../pulsar/functions/runtime/ProcessRuntime.java | 57 +- .../functions/runtime/ProcessRuntimeFactory.java | 3 +- .../apache/pulsar/functions/runtime/Runtime.java | 6 +- .../pulsar/functions/runtime/RuntimeSpawner.java | 4 +- .../pulsar/functions/runtime/ThreadRuntime.java | 27 +- .../functions/runtime/ProcessRuntimeTest.java | 13 +- .../apache/pulsar/functions/utils/Resources.java | 1 + .../apache/pulsar/functions/utils/SinkConfig.java | 8 +- .../pulsar/functions/utils/SourceConfig.java | 8 +- .../org/apache/pulsar/functions/utils/Utils.java | 21 +- .../utils/functioncache/FunctionCacheEntry.java | 16 +- .../utils/functioncache/FunctionCacheManager.java | 2 + .../functioncache/FunctionCacheManagerImpl.java | 24 + .../pulsar/functions/utils/io/ConnectorUtils.java | 153 ++++ .../utils/{Resources.java => io/Connectors.java} | 26 +- .../functions/utils/validation/ValidatorImpls.java | 209 +++-- pulsar-functions/worker/pom.xml | 15 + .../pulsar/functions/worker/ConnectorsManager.java | 48 + .../pulsar/functions/worker/FunctionActioner.java | 153 +++- .../functions/worker/FunctionRuntimeManager.java | 47 +- .../functions/worker/FunctionsStatsGenerator.java | 4 +- .../pulsar/functions/worker/MembershipManager.java | 30 - .../pulsar/functions/worker/WorkerConfig.java | 33 +- .../pulsar/functions/worker/WorkerService.java | 30 +- .../functions/worker/rest/api/FunctionsImpl.java | 509 +++++------ .../worker/rest/api/v2/FunctionApiV2Resource.java | 32 +- .../functions/worker/FunctionActionerTest.java | 8 +- .../worker/FunctionRuntimeManagerTest.java | 9 +- .../worker/FunctionStatsGeneratorTest.java | 3 +- .../functions/worker/MembershipManagerTest.java | 27 +- .../functions/worker/SchedulerManagerTest.java | 73 +- .../rest/api/v2/FunctionApiV2ResourceTest.java | 31 +- pulsar-io/aerospike/pom.xml | 18 +- .../pulsar/io/aerospike/AerospikeAbstractSink.java | 55 +- .../pulsar/io/aerospike/AerospikeStringSink.java | 6 +- .../resources/META-INF/services/pulsar-io.yaml | 6 +- pulsar-io/cassandra/pom.xml | 10 +- .../pulsar/io/cassandra/CassandraAbstractSink.java | 26 +- .../pulsar/io/cassandra/CassandraStringSink.java | 6 +- .../resources/META-INF/services/pulsar-io.yaml | 6 +- pulsar-io/core/pom.xml | 19 + .../java/org/apache/pulsar/io/core/PushSource.java | 6 +- .../java/org/apache/pulsar/io/core/SimpleSink.java | 46 - .../main/java/org/apache/pulsar/io/core/Sink.java | 15 +- .../io/core/{Record.java => SinkContext.java} | 13 +- .../java/org/apache/pulsar/io/core/Source.java | 5 +- .../io/core/{Record.java => SourceContext.java} | 13 +- .../java/org/apache/pulsar/io/core/SinkTest.java | 61 ++ .../java/org/apache/pulsar/io/core/SourceTest.java | 60 ++ pulsar-io/kafka/pom.xml | 10 +- .../apache/pulsar/io/kafka/KafkaAbstractSink.java | 53 +- .../pulsar/io/kafka/KafkaAbstractSource.java | 33 +- .../apache/pulsar/io/kafka/KafkaStringSink.java | 5 +- .../apache/pulsar/io/kafka/KafkaStringSource.java | 2 +- .../resources/META-INF/services/pulsar-io.yaml | 7 +- pulsar-io/kinesis/pom.xml | 61 +- .../kinesis/src/main/fb/KinesisMessageApi.fbs | 59 ++ .../org/apache/pulsar/io/kinesis/KinesisSink.java | 113 ++- .../pulsar/io/kinesis/KinesisSinkConfig.java | 30 +- .../java/org/apache/pulsar/io/kinesis/Utils.java | 221 +++++ .../pulsar/io/kinesis/fbs/CompressionType.java | 15 + .../pulsar/io/kinesis/fbs/EncryptionCtx.java | 68 ++ .../pulsar/io/kinesis/fbs/EncryptionKey.java | 52 ++ .../org/apache/pulsar/io/kinesis/fbs/KeyValue.java | 41 + .../org/apache/pulsar/io/kinesis/fbs/Message.java | 53 ++ .../resources/META-INF/services/pulsar-io.yaml | 6 +- .../org/apache/pulsar/io/kinesis/UtilsTest.java | 273 ++++++ pulsar-io/rabbitmq/pom.xml | 11 +- .../apache/pulsar/io/rabbitmq/RabbitMQSource.java | 31 +- .../resources/META-INF/services/pulsar-io.yaml | 6 +- pulsar-io/twitter/pom.xml | 11 +- .../apache/pulsar/io/twitter/TwitterFireHose.java | 20 +- .../resources/META-INF/services/pulsar-io.yaml | 6 +- pulsar-proxy/pom.xml | 2 +- .../pulsar/proxy/server/AdminProxyHandler.java | 10 + .../pulsar/proxy/server/LookupProxyHandler.java | 104 +++ .../pulsar/proxy/server/ProxyConnection.java | 8 + .../apache/pulsar/proxy/server/ProxyService.java | 12 +- .../pulsar/proxy/server/ProxyServiceStarter.java | 7 +- .../org/apache/pulsar/proxy/server/WebServer.java | 19 +- .../proxy/server/AuthedAdminProxyHandlerTest.java | 171 ++-- .../ProxyAuthenticatedProducerConsumerTest.java | 5 +- .../proxy/server/ProxyAuthenticationTest.java | 6 +- .../server/ProxyConnectionThrottlingTest.java | 5 +- .../proxy/server/ProxyForwardAuthDataTest.java | 10 +- .../proxy/server/ProxyLookupThrottlingTest.java | 7 +- .../proxy/server/ProxyRolesEnforcementTest.java | 6 +- .../org/apache/pulsar/proxy/server/ProxyTest.java | 54 +- .../apache/pulsar/proxy/server/ProxyTlsTest.java | 5 +- .../server/ProxyWithAuthorizationNegTest.java | 6 +- .../proxy/server/ProxyWithAuthorizationTest.java | 12 +- .../server/ProxyWithoutServiceDiscoveryTest.java | 6 +- .../SuperUserAuthedAdminProxyHandlerTest.java | 183 ++++ .../server/UnauthedAdminProxyHandlerTest.java | 5 +- .../authentication/tls-admin-proxy}/admin.cert.pem | 0 .../tls-admin-proxy}/admin.key-pk8.pem | 0 .../tls-admin-proxy}/broker.cert.pem | 0 .../tls-admin-proxy}/broker.key-pk8.pem | 0 .../authentication/tls-admin-proxy}/ca.cert.pem | 0 .../authentication/tls-admin-proxy/proxy.cert.pem | 26 + .../tls-admin-proxy/proxy.key-pk8.pem | 28 + .../tls-admin-proxy/randouser.cert.pem | 19 + .../tls-admin-proxy/randouser.key-pk8.pem | 28 + .../tls-admin-proxy/superproxy.cert.pem | 26 + .../tls-admin-proxy/superproxy.key-pk8.pem | 28 + .../authentication/tls-admin-proxy/user1.cert.pem | 26 + .../tls-admin-proxy/user1.key-pk8.pem | 28 + pulsar-spark/pom.xml | 2 +- pulsar-storm/pom.xml | 2 +- pulsar-testclient/pom.xml | 2 +- pulsar-websocket/pom.xml | 2 +- .../pulsar/websocket/WebSocketConsumerServlet.java | 6 +- .../pulsar/websocket/WebSocketProducerServlet.java | 5 +- .../pulsar/websocket/WebSocketReaderServlet.java | 6 +- .../service/WebSocketProxyConfiguration.java | 10 + pulsar-zookeeper-utils/pom.xml | 4 +- site/_data/cli/pulsar-admin.yaml | 158 ++++ .../_data/connectors.yaml | 41 +- site/_data/messages.yaml | 2 +- site/_data/popovers.yaml | 2 +- site/_data/sidebar.yaml | 31 +- .../pom.xml => site/_includes/connectors.html | 44 +- site/_includes/explanations/admin-setup.md | 4 +- site/_includes/explanations/client-url.md | 2 +- site/_includes/explanations/install-package.md | 2 +- site/_includes/explanations/tenants-namespaces.md | 2 +- site/_includes/figure.html | 9 +- site/_sass/_docs.scss | 18 + site/docs/latest/admin-api/brokers.md | 18 +- site/docs/latest/admin-api/clusters.md | 16 +- site/docs/latest/admin-api/namespaces.md | 100 +-- .../docs/latest/admin-api/non-persistent-topics.md | 10 +- site/docs/latest/admin-api/persistent-topics.md | 30 +- site/docs/latest/admin/Authz.md | 329 ------- site/docs/latest/admin/Dashboard.md | 16 +- site/docs/latest/admin/GeoReplication.md | 4 +- site/docs/latest/admin/ZooKeeperBookKeeper.md | 8 +- site/docs/latest/clients/Java.md | 6 +- site/docs/latest/clients/WebSocket.md | 18 +- site/docs/latest/clients/go.md | 6 +- site/docs/latest/cookbooks/RetentionExpiry.md | 28 +- site/docs/latest/cookbooks/compaction.md | 4 +- .../docs/latest/cookbooks/message-deduplication.md | 8 +- site/docs/latest/deployment/Kubernetes.md | 4 +- site/docs/latest/deployment/cluster.md | 8 +- site/docs/latest/deployment/instance.md | 2 +- site/docs/latest/functions/overview.md | 2 +- site/docs/latest/functions/quickstart.md | 6 +- .../getting-started/ConceptsAndArchitecture.md | 12 +- site/docs/latest/getting-started/docker.md | 2 +- site/docs/latest/io/overview.md | 51 ++ site/docs/latest/io/quickstart.md | 84 ++ site/docs/latest/project/BinaryProtocol.md | 11 +- site/docs/latest/security/athenz.md | 104 +++ site/docs/latest/security/authorization.md | 109 +++ .../Encryption.md => security/encryption.md} | 0 .../CustomAuth.md => security/extending.md} | 2 +- site/docs/latest/security/overview.md | 57 ++ site/docs/latest/security/tls.md | 161 ++++ site/img/pulsar-io.png | Bin 0 -> 37316 bytes src/stage-release.sh | 5 +- .../client-keys/proxy.cert.pem | 26 + .../client-keys/proxy.csr.pem | 15 + .../client-keys/proxy.key-pk8.pem | 28 + .../client-keys/proxy.key.pem | 27 + .../client-keys/randouser.cert.pem | 19 + .../client-keys/randouser.key-pk8.pem | 28 + .../client-keys/randouser.key.pem | 28 + .../client-keys/superproxy.cert.pem | 26 + .../client-keys/superproxy.csr.pem | 15 + .../client-keys/superproxy.key-pk8.pem | 28 + .../client-keys/superproxy.key.pem | 27 + .../client-keys/user1.cert.pem | 26 + .../client-keys/user1.csr.pem | 15 + .../client-keys/user1.key-pk8.pem | 28 + .../client-keys/user1.key.pem | 27 + tests/certificate-authority/index.txt | 4 + tests/certificate-authority/index.txt.old | 4 + tests/certificate-authority/newcerts/1002.pem | 26 + tests/certificate-authority/newcerts/1003.pem | 26 + tests/certificate-authority/newcerts/1004.pem | 26 + tests/certificate-authority/newcerts/1005.pem | 26 + tests/certificate-authority/serial | 2 +- tests/certificate-authority/serial.old | 2 +- .../docker-images/latest-version-image/Dockerfile | 12 +- .../conf/functions_worker.conf | 17 +- tests/docker-images/latest-version-image/pom.xml | 48 +- .../scripts/install-pulsar-client.sh | 8 +- .../scripts/run-functions-worker.sh | 17 +- tests/integration-tests-topologies/pom.xml | 16 + .../pulsar/tests/containers/BKContainer.java | 19 +- .../pulsar/tests/containers/BrokerContainer.java | 19 +- .../pulsar/tests/containers/CSContainer.java | 24 +- .../tests/containers/CassandraContainer.java | 52 ++ .../pulsar/tests/containers/ChaosContainer.java | 135 +++ .../pulsar/tests/containers/ProxyContainer.java | 27 +- .../pulsar/tests/containers/PulsarContainer.java | 124 +++ .../pulsar/tests/containers/S3Container.java | 54 ++ .../pulsar/tests/containers/WorkerContainer.java | 19 +- .../pulsar/tests/containers/ZKContainer.java | 72 ++ .../pulsar/tests/containers/package-info.java | 11 +- .../tests/topologies/FunctionRuntimeType.java | 12 +- .../pulsar/tests/topologies/PulsarCluster.java | 354 ++++++++ .../pulsar/tests/topologies/PulsarClusterSpec.java | 103 +++ .../tests/topologies/PulsarClusterTestBase.java | 144 +++ .../src/main/resources/kafka-zookeeper.properties | 27 +- tests/integration-tests-utils/pom.xml | 7 - .../java/org/apache/pulsar/tests/DockerUtils.java | 26 +- tests/integration/cli/pom.xml | 47 - .../apache/pulsar/tests/integration/TestCLI.java | 186 ---- tests/integration/compaction/pom.xml | 17 +- .../pulsar/tests/integration/TestCompaction.java | 140 ++- .../compaction/src/test/resources/arquillian.xml | 32 - tests/integration/pom.xml | 52 +- tests/integration/s3-offload/pom.xml | 11 +- .../pulsar/tests/integration/TestS3Offload.java | 228 +++-- .../s3-offload/src/test/resources/arquillian.xml | 32 - tests/integration/{smoke => semantics}/pom.xml | 34 +- .../pulsar/tests/integration/cli/CLITest.java | 167 ++++ .../integration/functions/PulsarFunctionsTest.java | 88 ++ .../functions/PulsarFunctionsTestBase.java | 90 ++ .../runtime/PulsarFunctionsRuntimeTest.java | 188 ++++ .../functions/utils/CommandGenerator.java | 260 ++++++ .../utils/UploadDownloadCommandGenerator.java | 74 ++ .../tests/integration/io/CassandraSinkTester.java | 118 +++ .../tests/integration/io/KafkaSinkTester.java | 130 +++ .../tests/integration/io/KafkaSourceTester.java | 140 +++ .../tests/integration/io/PulsarIOSinkTest.java | 257 ++++++ .../tests/integration/io/PulsarIOSourceTest.java | 255 ++++++ .../pulsar/tests/integration/io/SinkTester.java | 46 +- .../pulsar/tests/integration/io/SourceTester.java | 52 ++ .../tests/integration/semantics/SemanticsTest.java | 319 +++++++ .../pulsar/tests/integration/smoke/SmokeTest.java | 83 ++ .../pulsar/tests/integration/utils/TestUtils.java | 22 +- .../apache/pulsar/tests/integration/TestSmoke.java | 98 -- .../smoke/src/test/resources/arquillian.xml | 33 - 527 files changed, 19397 insertions(+), 4806 deletions(-) rename tests/integration/cli/src/test/resources/arquillian.xml => deployment/kubernetes/helm/README.md (60%) create mode 100644 deployment/kubernetes/helm/pulsar/.helmignore copy pulsar-functions/python-examples/pure_python_function_exclamation.py => deployment/kubernetes/helm/pulsar/Chart.yaml (87%) mode change 100755 => 100644 create mode 100644 deployment/kubernetes/helm/pulsar/templates/_helpers.tpl create mode 100644 deployment/kubernetes/helm/pulsar/templates/autorecovery-configmap.yaml create mode 100644 deployment/kubernetes/helm/pulsar/templates/autorecovery-deployment.yaml copy pulsar-functions/java-examples/src/main/resources/example-stateful-function-config.yaml => deployment/kubernetes/helm/pulsar/templates/bastion-configmap.yaml (60%) create mode 100644 deployment/kubernetes/helm/pulsar/templates/bastion-deployment.yaml create mode 100644 deployment/kubernetes/helm/pulsar/templates/bookkeeper-configmap.yaml copy pulsar-client-cpp/docker/build-wheels.sh => deployment/kubernetes/helm/pulsar/templates/bookkeeper-pdb.yaml (52%) mode change 100755 => 100644 copy pulsar-client-cpp/docker/build-wheels.sh => deployment/kubernetes/helm/pulsar/templates/bookkeeper-service.yaml (52%) mode change 100755 => 100644 create mode 100644 deployment/kubernetes/helm/pulsar/templates/bookkeeper-statefulset.yaml create mode 100644 deployment/kubernetes/helm/pulsar/templates/bookkeeper-storageclass.yaml create mode 100644 deployment/kubernetes/helm/pulsar/templates/broker-configmap.yaml create mode 100644 deployment/kubernetes/helm/pulsar/templates/broker-deployment.yaml copy pulsar-client-cpp/docker/build-wheels.sh => deployment/kubernetes/helm/pulsar/templates/broker-pdb.yaml (52%) mode change 100755 => 100644 copy pulsar-client-cpp/docker/build-wheels.sh => deployment/kubernetes/helm/pulsar/templates/broker-service.yaml (53%) mode change 100755 => 100644 create mode 100644 deployment/kubernetes/helm/pulsar/templates/dashboard-deployment.yaml copy pulsar-client-cpp/docker/build-wheels.sh => deployment/kubernetes/helm/pulsar/templates/dashboard-service.yaml (50%) mode change 100755 => 100644 create mode 100644 deployment/kubernetes/helm/pulsar/templates/grafana-deployment.yaml copy pulsar-client-cpp/docker/build-wheels.sh => deployment/kubernetes/helm/pulsar/templates/grafana-service.yaml (51%) mode change 100755 => 100644 copy pulsar-functions/python-examples/pure_python_function_exclamation.py => deployment/kubernetes/helm/pulsar/templates/namespace.yaml (87%) mode change 100755 => 100644 create mode 100644 deployment/kubernetes/helm/pulsar/templates/prometheus-configmap.yaml create mode 100644 deployment/kubernetes/helm/pulsar/templates/prometheus-deployment.yaml copy dashboard/conf/supervisor-app.conf => deployment/kubernetes/helm/pulsar/templates/prometheus-pvc.yaml (56%) create mode 100644 deployment/kubernetes/helm/pulsar/templates/prometheus-rbac.yaml copy pulsar-client-cpp/docker/build-wheels.sh => deployment/kubernetes/helm/pulsar/templates/prometheus-service.yaml (50%) mode change 100755 => 100644 create mode 100644 deployment/kubernetes/helm/pulsar/templates/prometheus-storageclass.yaml create mode 100644 deployment/kubernetes/helm/pulsar/templates/proxy-configmap.yaml create mode 100644 deployment/kubernetes/helm/pulsar/templates/proxy-deployment.yaml copy pulsar-client-cpp/docker/build-wheels.sh => deployment/kubernetes/helm/pulsar/templates/proxy-pdb.yaml (51%) mode change 100755 => 100644 copy pulsar-client-cpp/docker/build-wheels.sh => deployment/kubernetes/helm/pulsar/templates/proxy-service.yaml (51%) mode change 100755 => 100644 copy pulsar-functions/java-examples/src/main/resources/example-stateful-function-config.yaml => deployment/kubernetes/helm/pulsar/templates/zookeeper-configmap.yaml (62%) create mode 100644 deployment/kubernetes/helm/pulsar/templates/zookeeper-metadata.yaml copy pulsar-client-cpp/docker/build-wheels.sh => deployment/kubernetes/helm/pulsar/templates/zookeeper-pdb.yaml (52%) mode change 100755 => 100644 copy pulsar-client-cpp/docker/build-wheels.sh => deployment/kubernetes/helm/pulsar/templates/zookeeper-service.yaml (52%) mode change 100755 => 100644 create mode 100644 deployment/kubernetes/helm/pulsar/templates/zookeeper-statefulset.yaml create mode 100644 deployment/kubernetes/helm/pulsar/templates/zookeeper-storageclass.yaml create mode 100644 deployment/kubernetes/helm/pulsar/values.yaml copy {pulsar-io/kafka => distribution/io}/pom.xml (56%) create mode 100644 distribution/io/src/assemble/README create mode 100644 distribution/io/src/assemble/io.xml copy {pulsar-io/core => distribution}/pom.xml (79%) rename {all => distribution/server}/licenses/LICENSE-AspectJ.txt (100%) rename {all => distribution/server}/licenses/LICENSE-CC0.txt (100%) rename {all => distribution/server}/licenses/LICENSE-CDDL-1.1.txt (100%) rename {all => distribution/server}/licenses/LICENSE-EA-Agent-Loader.txt (100%) rename {all => distribution/server}/licenses/LICENSE-EPL-1.0.txt (100%) rename {all => distribution/server}/licenses/LICENSE-Hamcrest.txt (100%) rename {all => distribution/server}/licenses/LICENSE-HdrHistogram.txt (100%) rename {all => distribution/server}/licenses/LICENSE-HdrHistogram0.txt (100%) rename {all => distribution/server}/licenses/LICENSE-JSR305.txt (100%) rename {all => distribution/server}/licenses/LICENSE-LevelDB.txt (100%) rename {all => distribution/server}/licenses/LICENSE-Lombok.txt (100%) rename {all => distribution/server}/licenses/LICENSE-SLF4J.txt (100%) rename {all => distribution/server}/licenses/LICENSE-SemVer.txt (100%) rename {all => distribution/server}/licenses/LICENSE-Webbit.txt (100%) rename {all => distribution/server}/licenses/LICENSE-bouncycastle.txt (100%) rename {all => distribution/server}/licenses/LICENSE-google-auth-library.txt (100%) rename {all => distribution/server}/licenses/LICENSE-jbzip2.txt (100%) rename {all => distribution/server}/licenses/LICENSE-jfastlz.txt (100%) rename {all => distribution/server}/licenses/LICENSE-libdivsufsort.txt (100%) rename {all => distribution/server}/licenses/LICENSE-protobuf.txt (100%) rename {all => distribution/server}/licenses/LICENSE-xz.txt (100%) rename {all => distribution/server}/pom.xml (97%) rename {all => distribution/server}/src/assemble/LICENSE.bin.txt (99%) rename {all => distribution/server}/src/assemble/NOTICE.bin.txt (100%) rename {all => distribution/server}/src/assemble/README.bin.txt (100%) rename {all => distribution/server}/src/assemble/bin.xml (83%) rename {all => distribution/server}/src/assemble/src.xml (99%) copy pulsar-functions/python-examples/pure_python_function_exclamation.py => docker/pulsar-all/Dockerfile (83%) mode change 100755 => 100644 copy docker/{pulsar => pulsar-all}/pom.xml (73%) create mode 100755 docker/pulsar/scripts/gen-yml-from-env.py create mode 100644 managed-ledger-shaded/pom.xml create mode 100644 managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ReadOnlyCursor.java create mode 100644 managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ReadOnlyCursorImpl.java create mode 100644 managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ReadOnlyManagedLedgerImpl.java create mode 100644 managed-ledger/src/test/java/org/apache/bookkeeper/mledger/impl/ReadOnlyCursorTest.java copy pulsar-broker/src/main/java/org/apache/pulsar/{PulsarStandaloneStarter.java => PulsarStandalone.java} (64%) create mode 100644 pulsar-broker/src/main/java/org/apache/pulsar/PulsarStandaloneBuilder.java rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload/S3BackedInputStream.java => offload/BackedInputStream.java} (90%) rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload => offload}/BlockAwareSegmentInputStream.java (96%) rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload => offload}/DataBlockHeader.java (95%) rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload => offload}/OffloadIndexBlock.java (98%) rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload => offload}/OffloadIndexBlockBuilder.java (95%) rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload => offload}/OffloadIndexEntry.java (97%) rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload => offload}/impl/BlockAwareSegmentInputStreamImpl.java (98%) rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload => offload}/impl/DataBlockHeaderImpl.java (97%) rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload => offload}/impl/OffloadIndexBlockBuilderImpl.java (94%) rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload => offload}/impl/OffloadIndexBlockImpl.java (98%) rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload => offload}/impl/OffloadIndexEntryImpl.java (94%) rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload => offload}/impl/S3BackedInputStreamImpl.java (95%) rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload => offload}/impl/S3BackedReadHandleImpl.java (92%) rename pulsar-broker/src/main/java/org/apache/pulsar/broker/{s3offload => offload/impl}/S3ManagedLedgerOffloader.java (98%) create mode 100644 pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiTlsAuthTest.java rename pulsar-broker/src/test/java/org/apache/pulsar/broker/{s3offload => offload}/S3BackedInputStreamTest.java (91%) rename pulsar-broker/src/test/java/org/apache/pulsar/broker/{s3offload => offload}/S3Mock.java (99%) rename pulsar-broker/src/test/java/org/apache/pulsar/broker/{s3offload => offload}/S3TestBase.java (84%) rename pulsar-broker/src/test/java/org/apache/pulsar/broker/{s3offload => offload}/impl/BlockAwareSegmentInputStreamTest.java (99%) rename pulsar-broker/src/test/java/org/apache/pulsar/broker/{s3offload => offload}/impl/DataBlockHeaderTest.java (96%) rename pulsar-broker/src/test/java/org/apache/pulsar/broker/{s3offload => offload}/impl/OffloadIndexTest.java (97%) rename pulsar-broker/src/test/java/org/apache/pulsar/broker/{s3offload => offload/impl}/S3ManagedLedgerOffloaderTest.java (98%) copy pulsar-broker/src/test/java/org/apache/pulsar/broker/service/schema/{AvroSchemaCompatibilityCheckTest.java => BaseAvroSchemaCompatibilityTest.java} (70%) create mode 100644 pulsar-broker/src/test/java/org/apache/pulsar/broker/service/schema/JsonSchemaCompatibilityCheckTest.java copy pulsar-broker/src/test/java/org/apache/pulsar/{client/impl/SequenceIdWithErrorTest.java => broker/service/schema/PartitionedTopicsSchemaTest.java} (54%) copy pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/BytesSchema.java => pulsar-broker/src/test/java/org/apache/pulsar/broker/service/schema/ProtobufSchemaCompatibilityCheckTest.java (58%) create mode 100644 pulsar-broker/src/test/java/org/apache/pulsar/client/impl/MessageParserTest.java copy pulsar-broker/src/test/java/org/apache/pulsar/io/{PulsarSinkE2ETest.java => PulsarFunctionAdminTest.java} (61%) copy {tests/docker-images/latest-version-image/ssl => pulsar-broker/src/test/resources/authentication/tls-http}/admin.cert.pem (100%) copy {tests/docker-images/latest-version-image/ssl => pulsar-broker/src/test/resources/authentication/tls-http}/admin.key-pk8.pem (100%) copy {tests/docker-images/latest-version-image/ssl => pulsar-broker/src/test/resources/authentication/tls-http}/broker.cert.pem (100%) copy {tests/docker-images/latest-version-image/ssl => pulsar-broker/src/test/resources/authentication/tls-http}/broker.key-pk8.pem (100%) copy {tests/docker-images/latest-version-image/ssl => pulsar-broker/src/test/resources/authentication/tls-http}/ca.cert.pem (100%) create mode 100644 pulsar-broker/src/test/resources/authentication/tls-http/proxy.cert.pem create mode 100644 pulsar-broker/src/test/resources/authentication/tls-http/proxy.key-pk8.pem create mode 100644 pulsar-broker/src/test/resources/authentication/tls-http/superproxy.cert.pem create mode 100644 pulsar-broker/src/test/resources/authentication/tls-http/superproxy.key-pk8.pem create mode 100644 pulsar-client-schema/pom.xml rename {pulsar-client => pulsar-client-schema}/src/main/java/org/apache/pulsar/client/api/Schema.java (81%) copy {pulsar-client => pulsar-client-schema}/src/main/java/org/apache/pulsar/client/api/SchemaSerializationException.java (100%) rename {pulsar-client => pulsar-client-schema}/src/main/java/org/apache/pulsar/client/impl/schema/AvroSchema.java (97%) copy {pulsar-client => pulsar-client-schema}/src/main/java/org/apache/pulsar/client/impl/schema/BytesSchema.java (100%) rename {pulsar-client => pulsar-client-schema}/src/main/java/org/apache/pulsar/client/impl/schema/JSONSchema.java (52%) create mode 100644 pulsar-client-schema/src/main/java/org/apache/pulsar/client/impl/schema/ProtobufSchema.java rename {pulsar-client => pulsar-client-schema}/src/main/java/org/apache/pulsar/client/impl/schema/StringSchema.java (100%) copy {pulsar-client/src/test/java/org/apache/pulsar/client/schemas => pulsar-client-schema/src/test/java/org/apache/pulsar/client/schema}/AvroSchemaTest.java (95%) rename pulsar-client/src/test/java/org/apache/pulsar/client/schemas/AvroSchemaTest.java => pulsar-client-schema/src/test/java/org/apache/pulsar/client/schema/JSONSchemaTest.java (79%) rename {pulsar-client/src/test/java/org/apache/pulsar/client/schemas => pulsar-client-schema/src/test/java/org/apache/pulsar/client/schema}/ProtobufSchemaTest.java (50%) rename {pulsar-client => pulsar-client-schema}/src/test/proto/Test.proto (89%) create mode 100644 pulsar-client-tools/src/test/java/org/apache/pulsar/admin/cli/TestCmdSinks.java create mode 100644 pulsar-client-tools/src/test/java/org/apache/pulsar/admin/cli/TestCmdSources.java create mode 100644 pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageParser.java delete mode 100644 pulsar-client/src/main/java/org/apache/pulsar/client/impl/MessageRecordImpl.java delete mode 100644 pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/ProtobufSchema.java delete mode 100644 pulsar-client/src/test/java/org/apache/pulsar/client/impl/MessageRecordImplTest.java rename pulsar-client/src/test/java/org/apache/pulsar/client/{schemas => schema}/DefaultSchemasTest.java (88%) copy pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/PulsarRecord.java => pulsar-common/src/main/java/org/apache/pulsar/common/api/EncryptionContext.java (54%) copy pulsar-io/core/src/main/java/org/apache/pulsar/io/core/RecordContext.java => pulsar-common/src/main/java/org/apache/pulsar/common/io/ConnectorDefinition.java (57%) create mode 100644 pulsar-common/src/main/java/org/apache/pulsar/common/nar/FileUtils.java create mode 100644 pulsar-common/src/main/java/org/apache/pulsar/common/nar/NarClassLoader.java create mode 100644 pulsar-common/src/main/java/org/apache/pulsar/common/nar/NarUnpacker.java rename pulsar-io/core/src/main/java/org/apache/pulsar/io/core/RecordContext.java => pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Record.java (56%) create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/SinkRecord.java copy pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Record.java => pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/source/RecordWithEncryptionContext.java (66%) rename pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/{UserConfigFunction.java => ConfigBasedAppendFunction.java} (68%) copy pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/{CounterFunction.java => CustomObjectFunction.java} (74%) rename pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/{CounterFunction.java => WordCountFunction.java} (78%) copy pulsar-functions/{utils/src/main/java/org/apache/pulsar/functions/utils/Resources.java => java-examples/src/main/java/org/apache/pulsar/functions/api/examples/serde/CustomObject.java} (78%) copy pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/BytesSchema.java => pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/serde/CustomObjectSerde.java (58%) copy pulsar-functions/{instance/src/main/java/org/apache/pulsar/functions/source/PulsarRecord.java => proto/src/main/java/org/apache/pulsar/functions/worker/WorkerInfo.java} (50%) copy pulsar-functions/python-examples/{logfunction.py => config_based_append_function.py} (75%) copy pulsar-functions/python-examples/{exclamation.py => custom_object_function.py} (59%) copy pulsar-functions/python-examples/{logfunction.py => exclamation_function.py} (88%) copy pulsar-functions/python-examples/{exclamation.py => logging_function.py} (73%) copy pulsar-functions/python-examples/{pure_python_function_exclamation.py => native_exclamation_function.py} (100%) copy pulsar-functions/python-examples/{logfunction.py => publish_function.py} (70%) rename pulsar-functions/python-examples/{exclamation.py => void_function.py} (90%) create mode 100644 pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/io/ConnectorUtils.java copy pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/{Resources.java => io/Connectors.java} (66%) create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/ConnectorsManager.java copy pulsar-functions/python-examples/pure_python_function_exclamation.py => pulsar-io/aerospike/src/main/resources/META-INF/services/pulsar-io.yaml (87%) mode change 100755 => 100644 copy pulsar-functions/python-examples/pure_python_function_exclamation.py => pulsar-io/cassandra/src/main/resources/META-INF/services/pulsar-io.yaml (87%) mode change 100755 => 100644 delete mode 100644 pulsar-io/core/src/main/java/org/apache/pulsar/io/core/SimpleSink.java copy pulsar-io/core/src/main/java/org/apache/pulsar/io/core/{Record.java => SinkContext.java} (75%) rename pulsar-io/core/src/main/java/org/apache/pulsar/io/core/{Record.java => SourceContext.java} (75%) create mode 100644 pulsar-io/core/src/test/java/org/apache/pulsar/io/core/SinkTest.java create mode 100644 pulsar-io/core/src/test/java/org/apache/pulsar/io/core/SourceTest.java copy pulsar-functions/python-examples/pure_python_function_exclamation.py => pulsar-io/kafka/src/main/resources/META-INF/services/pulsar-io.yaml (82%) mode change 100755 => 100644 create mode 100644 pulsar-io/kinesis/src/main/fb/KinesisMessageApi.fbs create mode 100644 pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/Utils.java create mode 100644 pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/fbs/CompressionType.java create mode 100644 pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/fbs/EncryptionCtx.java create mode 100644 pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/fbs/EncryptionKey.java create mode 100644 pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/fbs/KeyValue.java create mode 100644 pulsar-io/kinesis/src/main/java/org/apache/pulsar/io/kinesis/fbs/Message.java copy pulsar-functions/python-examples/pure_python_function_exclamation.py => pulsar-io/kinesis/src/main/resources/META-INF/services/pulsar-io.yaml (88%) mode change 100755 => 100644 create mode 100644 pulsar-io/kinesis/src/test/java/org/apache/pulsar/io/kinesis/UtilsTest.java copy pulsar-functions/python-examples/pure_python_function_exclamation.py => pulsar-io/rabbitmq/src/main/resources/META-INF/services/pulsar-io.yaml (87%) mode change 100755 => 100644 copy pulsar-functions/python-examples/pure_python_function_exclamation.py => pulsar-io/twitter/src/main/resources/META-INF/services/pulsar-io.yaml (86%) mode change 100755 => 100644 create mode 100644 pulsar-proxy/src/test/java/org/apache/pulsar/proxy/server/SuperUserAuthedAdminProxyHandlerTest.java copy {tests/docker-images/latest-version-image/ssl => pulsar-proxy/src/test/resources/authentication/tls-admin-proxy}/admin.cert.pem (100%) copy {tests/docker-images/latest-version-image/ssl => pulsar-proxy/src/test/resources/authentication/tls-admin-proxy}/admin.key-pk8.pem (100%) copy {tests/docker-images/latest-version-image/ssl => pulsar-proxy/src/test/resources/authentication/tls-admin-proxy}/broker.cert.pem (100%) copy {tests/docker-images/latest-version-image/ssl => pulsar-proxy/src/test/resources/authentication/tls-admin-proxy}/broker.key-pk8.pem (100%) copy {tests/docker-images/latest-version-image/ssl => pulsar-proxy/src/test/resources/authentication/tls-admin-proxy}/ca.cert.pem (100%) create mode 100644 pulsar-proxy/src/test/resources/authentication/tls-admin-proxy/proxy.cert.pem create mode 100644 pulsar-proxy/src/test/resources/authentication/tls-admin-proxy/proxy.key-pk8.pem create mode 100644 pulsar-proxy/src/test/resources/authentication/tls-admin-proxy/randouser.cert.pem create mode 100644 pulsar-proxy/src/test/resources/authentication/tls-admin-proxy/randouser.key-pk8.pem create mode 100644 pulsar-proxy/src/test/resources/authentication/tls-admin-proxy/superproxy.cert.pem create mode 100644 pulsar-proxy/src/test/resources/authentication/tls-admin-proxy/superproxy.key-pk8.pem create mode 100644 pulsar-proxy/src/test/resources/authentication/tls-admin-proxy/user1.cert.pem create mode 100644 pulsar-proxy/src/test/resources/authentication/tls-admin-proxy/user1.key-pk8.pem copy pulsar-functions/java-examples/src/main/resources/example-stateful-function-config.yaml => site/_data/connectors.yaml (53%) copy tests/integration-tests-topologies/pom.xml => site/_includes/connectors.html (53%) delete mode 100644 site/docs/latest/admin/Authz.md create mode 100644 site/docs/latest/io/overview.md create mode 100644 site/docs/latest/io/quickstart.md create mode 100644 site/docs/latest/security/athenz.md create mode 100644 site/docs/latest/security/authorization.md copy site/docs/latest/{cookbooks/Encryption.md => security/encryption.md} (100%) rename site/docs/latest/{reference/CustomAuth.md => security/extending.md} (99%) create mode 100644 site/docs/latest/security/overview.md create mode 100644 site/docs/latest/security/tls.md create mode 100644 site/img/pulsar-io.png create mode 100644 tests/certificate-authority/client-keys/proxy.cert.pem create mode 100644 tests/certificate-authority/client-keys/proxy.csr.pem create mode 100644 tests/certificate-authority/client-keys/proxy.key-pk8.pem create mode 100644 tests/certificate-authority/client-keys/proxy.key.pem create mode 100644 tests/certificate-authority/client-keys/randouser.cert.pem create mode 100644 tests/certificate-authority/client-keys/randouser.key-pk8.pem create mode 100644 tests/certificate-authority/client-keys/randouser.key.pem create mode 100644 tests/certificate-authority/client-keys/superproxy.cert.pem create mode 100644 tests/certificate-authority/client-keys/superproxy.csr.pem create mode 100644 tests/certificate-authority/client-keys/superproxy.key-pk8.pem create mode 100644 tests/certificate-authority/client-keys/superproxy.key.pem create mode 100644 tests/certificate-authority/client-keys/user1.cert.pem create mode 100644 tests/certificate-authority/client-keys/user1.csr.pem create mode 100644 tests/certificate-authority/client-keys/user1.key-pk8.pem create mode 100644 tests/certificate-authority/client-keys/user1.key.pem create mode 100644 tests/certificate-authority/newcerts/1002.pem create mode 100644 tests/certificate-authority/newcerts/1003.pem create mode 100644 tests/certificate-authority/newcerts/1004.pem create mode 100644 tests/certificate-authority/newcerts/1005.pem copy pulsar-functions/python-examples/logfunction.py => tests/docker-images/latest-version-image/conf/functions_worker.conf (79%) mode change 100755 => 100644 rename pulsar-functions/python-examples/pure_python_function_exclamation.py => tests/docker-images/latest-version-image/scripts/install-pulsar-client.sh (87%) rename pulsar-functions/python-examples/logfunction.py => tests/docker-images/latest-version-image/scripts/run-functions-worker.sh (64%) copy pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaStringSource.java => tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/BKContainer.java (69%) copy pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaStringSource.java => tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/BrokerContainer.java (67%) copy pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaStringSource.java => tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/CSContainer.java (66%) create mode 100644 tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/CassandraContainer.java create mode 100644 tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/ChaosContainer.java rename pulsar-client/src/main/java/org/apache/pulsar/client/impl/schema/BytesSchema.java => tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/ProxyContainer.java (55%) create mode 100644 tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/PulsarContainer.java create mode 100644 tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/S3Container.java copy pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaStringSource.java => tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/WorkerContainer.java (67%) create mode 100644 tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/ZKContainer.java copy pulsar-client/src/main/java/org/apache/pulsar/client/api/SchemaSerializationException.java => tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/containers/package-info.java (80%) rename pulsar-client/src/main/java/org/apache/pulsar/client/api/SchemaSerializationException.java => tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/topologies/FunctionRuntimeType.java (80%) create mode 100644 tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/topologies/PulsarCluster.java create mode 100644 tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/topologies/PulsarClusterSpec.java create mode 100644 tests/integration-tests-topologies/src/main/java/org/apache/pulsar/tests/topologies/PulsarClusterTestBase.java copy dashboard/conf/supervisor-app.conf => tests/integration-tests-topologies/src/main/resources/kafka-zookeeper.properties (55%) delete mode 100644 tests/integration/cli/pom.xml delete mode 100644 tests/integration/cli/src/test/java/org/apache/pulsar/tests/integration/TestCLI.java delete mode 100644 tests/integration/compaction/src/test/resources/arquillian.xml delete mode 100644 tests/integration/s3-offload/src/test/resources/arquillian.xml rename tests/integration/{smoke => semantics}/pom.xml (60%) create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/cli/CLITest.java create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTest.java create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/PulsarFunctionsTestBase.java create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/runtime/PulsarFunctionsRuntimeTest.java create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/utils/CommandGenerator.java create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/functions/utils/UploadDownloadCommandGenerator.java create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/io/CassandraSinkTester.java create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/io/KafkaSinkTester.java create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/io/KafkaSourceTester.java create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/io/PulsarIOSinkTest.java create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/io/PulsarIOSourceTest.java copy pulsar-io/core/src/main/java/org/apache/pulsar/io/core/Sink.java => tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/io/SinkTester.java (50%) create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/io/SourceTester.java create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/semantics/SemanticsTest.java create mode 100644 tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/smoke/SmokeTest.java copy pulsar-io/kafka/src/main/java/org/apache/pulsar/io/kafka/KafkaStringSource.java => tests/integration/semantics/src/test/java/org/apache/pulsar/tests/integration/utils/TestUtils.java (66%) delete mode 100644 tests/integration/smoke/src/test/java/org/apache/pulsar/tests/integration/TestSmoke.java delete mode 100644 tests/integration/smoke/src/test/resources/arquillian.xml