cbornet opened a new pull request, #15903:
URL: https://github.com/apache/pulsar/pull/15903
<!--
### Contribution Checklist
- PR title format should be *[type][component] summary*. For details, see
*[Guideline - Pulsar PR Naming
Convention](https://docs.google.com/document/d/1d8Pw6ZbWk-_pCKdOmdvx9rnhPiyuxwq60_TrD68d7BA/edit#heading=h.trs9rsex3xom)*.
- Fill out the template below to describe the changes contributed by the
pull request. That will give reviewers the context they need to do the review.
- Each pull request should address only one issue, not mix up code from
multiple issues.
- Each commit in the pull request has a meaningful commit message
- Once all items of the checklist are addressed, remove the above text and
this checklist, leaving only the filled out template below.
**(The sections below can be removed for hotfixes of typos)**
-->
### Motivation
This PR implements [PIP-173 : Create a built-in Function implementing the
most common basic
transformations](https://github.com/apache/pulsar/issues/15902).
It provides a first set of basic functions:
* `cast`: modifies the key or value schema to a target compatible schema
passed in the `schema-type` argument. This PR only enables `STRING`
schema-type. The `part` argument allows to choose on which part to apply
between `key` and `value`. If `part` is null or absent the transformations
aplpies to both the key and value.
* `drop-fields`: drops fields given as a string list in parameter `fields`.
The `part` argument allows to choose on which part to apply between `key` and
`value`. If `part` is null or absent the transformations aplpies to both the
key and value.
* `merge-key-value`: merges the fields of KeyValue records where both the
key and value are structured types of the same schema type.
* `unwrap-key-value`: if the record is a KeyValue, extract the KeyValue's
value and make it the record value. If parameter `unwrapKey` is present and set
to `true`, extract the KeyValue's key instead.
### Modifications
Add a `transforms` module in multi-module `pulsar-function` project.
Create classes `TransformFunction`, `TransformContext`, interface
`TransformStep` described in PIP-173
Create implementations `CastStep`, `DropFieldsStep`, `MergeKeyValueStep`,
`UnwrapKeyValueStep` with tests
Add logic to parse `userConfig` in `TransformFunction` and call the
`TransformStep` implementations.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
Run tests from the `transforms` module
### Does this pull request potentially affect one of the following parts:
no
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
Check the box below or label this PR directly.
Need to update docs?
- [x] `doc-required`
(Your PR needs to update docs and you will update later)
- [ ] `doc-not-needed`
(Please explain why)
- [ ] `doc`
(Your PR contains doc changes)
- [ ] `doc-complete`
(Docs have been already added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
