devinbost opened a new pull request #10154: URL: https://github.com/apache/pulsar/pull/10154
Part 1 of [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults For details, see the [mailing list discussion](https://lists.apache.org/thread.html/rc58361593fbd20bb6ca714aed0d84ffbaf54c305c0824d3bfed0c8ef%40%3Cdev.pulsar.apache.org%3E) Fixes #9986 This PR adds these properties to `ProducerConfig`: ``` public Boolean batchingDisabled; public Boolean chunkingEnabled; public Boolean blockIfQueueFullDisabled; public CompressionType compressionType; public HashingScheme hashingScheme; public MessageRoutingMode messageRoutingMode; public Long batchingMaxPublishDelay; ``` It updates the methods that convert between the `ProducerConfig` and the protobuf `ProducerSpec`. It adds logic to use the new function producer settings when they are provided; otherwise, it reverts to the prior behavior. It also refactors the `FunctionResultRouter` to prevent a race condition on `isBatchingEnabled`. It adds these properties to `ProducerSpec` in `Function.proto`: ``` bool batchingDisabled = 6; bool chunkingEnabled = 7; bool blockIfQueueFullDisabled = 8; CompressionType compressionType = 9; HashingScheme hashingScheme = 10; MessageRoutingMode messageRoutingMode = 11; int64 batchingMaxPublishDelay = 12; ``` and adds these new enums to Function.proto: ``` enum CompressionType { LZ4 = 0; NONE = 1; ZLIB = 2; ZSTD = 3; SNAPPY = 4; } enum HashingScheme { MURMUR3_32HASH = 0; JAVA_STRING_HASH = 1; } enum MessageRoutingMode { CUSTOM_PARTITION = 0; SINGLE_PARTITION = 1; ROUND_ROBIN_PARTITION = 2; } ``` The new protobuf files have not yet been generated for Python and Go functions, but those changes are planned in a subsequent PR. Existing tests have been updated to include the new properties, and additional tests have been provided to ensure that new configs merge properly into existing configs during function registration/update. Changes did not need to be made to the Admin CLI parameters because the Admin CLI already provides a way for the `ProducerConfig` to be provided as a JSON string. Docs will be provided in a subsequent PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
