devinbost opened a new pull request #4012: Adding upsert functionality URL: https://github.com/apache/pulsar/pull/4012 ### Motivation Upsert functionality is intended to simplify continuous deployment processes. This simplification is especially desirable because there is no Python Admin API available that provides typed objects. Rather, users are required to build workflows with the Admin CLI, which returns strings that can be messy to handle. Upsert helps solve this complexity by allowing Pulsar to handle behavior according to whether the component has already been created or not. ### Motivation (with future in mind) Upsert is a dependency of BulkUpsert (to be added), which will greatly simplify seamless continuous deployments by allowing Pulsar to rapidly update Functions, Sinks, and Sources (and leverage parallelization). BulkUpsert will enable automated deployments to handle all but deletes on components that are no longer used. (For that, functionality will need to be added to allow the user to easily compare their expected component tree with the component tree in Pulsar to identify components that need to be deleted.) ### Modifications My contributions add Upsert functionality for Functions, Sinks, and Sources to the REST API and to the Admin CLI. Upsert conditionally combines the functionality of Create and Update operations. If the component already exists in Pulsar, Upsert updates the component; if the component doesn't already exist in Pulsar, Upsert creates the component. Commits 29e2a66 and 7749419 were just to get the latest changes from the Pulsar repo. My primary code is in commit 441ba03, and the unit tests are in commit 441ba03. ### Verifying this change This commit is largely covered already by unit tests for registration and updates for Functions, Sinks, and Sources. However, additional unit tests have been added for Upsert specifically. Tests were added to: - org.apache.pulsar.functions.worker/rest.api/FunctionApiV3ResourceTest.java - org.apache.pulsar.functions.worker/rest.api/SinkApiV3ResourceTest.java - org.apache.pulsar.functions.worker/rest.api/SourceApiV3ResourceTest.java - org.apache.pulsar.functions.worker/FunctionMetaDataManagerTest.java When running the build tests, I did notice these **unrelated** test failures: 1. > [ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 13.506 s <<< FAILURE! - in org.apache.pulsar.broker.service.RackAwareTest > [ERROR] testPlacement(org.apache.pulsar.broker.service.RackAwareTest) Time elapsed: 1.085 s <<< FAILURE! > java.lang.AssertionError: first bookie in rack 0 not included in ensemble expected [true] but found [false] 2. > [ERROR] Tests run: 13, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 59.026 s <<< FAILURE! - in org.apache.pulsar.io.PulsarFunctionE2ETest > [ERROR] testPulsarSinkStatsWithUrl(org.apache.pulsar.io.PulsarFunctionE2ETest) Time elapsed: 0.614 s <<< FAILURE! > java.lang.NullPointerException > at org.apache.pulsar.io.PulsarFunctionE2ETest.testPulsarSinkStats(PulsarFunctionE2ETest.java:527) > at org.apache.pulsar.io.PulsarFunctionE2ETest.testPulsarSinkStatsWithUrl(PulsarFunctionE2ETest.java:708) It appears that these test failures are unrelated to any changes that I made to this repo. However, I ran these tests before merging in commit 29e2a66. (Sorry if that's confusing. I was working in a different directory.) ### This pull request affects: - The public API: (**yes**) - The rest endpoints: (**yes**) - The admin cli options: (**yes**) - Anything that affects deployment: (don't know) ### Documentation - Does this pull request introduce a new feature? (**yes**) - If yes, how is the feature documented? (**not documented YET**). (Please provide guidance on where it needs to be documented. There are 930 markdown (.md) files in this repo...)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
