akhil-sabu-ibm opened a new pull request, #23084:
URL: https://github.com/apache/kafka/pull/23084
## Summary
This PR implements a new `TranslateSurrogates` transformation to handle
UTF-16 surrogate pairs in string fields.
## Changes
**Files:**
-
`connect/transforms/src/main/java/org/apache/kafka/connect/transforms/TranslateSurrogates.java`
(new)
-
`connect/transforms/src/main/resources/META-INF/services/org.apache.kafka.connect.transforms.Transformation`
(updated)
-
`connect/transforms/src/test/java/org/apache/kafka/connect/transforms/TranslateSurrogatesTest.java`
(new)
**Motivation:** Some data sources produce UTF-16 surrogate pairs in string
fields that cannot be properly serialized or processed by downstream systems.
This transformation provides configurable handling of such characters.
**Features:**
- Supports three encoding modes:
- `url-encode` (default): Percent-encodes using UTF-8 byte sequence (e.g.,
U+23A3A → `%F0%A3%98%BA`)
- `java-encode`: Java-style surrogate pair notation (e.g., `\UD84D\UDE3A`)
- `replace`: Replaces with configurable string (defaults to Unicode
replacement character U+FFFD)
- Processes string fields at any depth in structs, maps, and arrays
- Supports optional field filtering via `fields` config
- Available as both `Key` and `Value` transformation variants
- Validates that replacement strings do not themselves contain surrogate
pairs
## Testing Strategy
Comprehensive test suite covering:
- **Configuration validation**: Invalid modes, replacement strings
containing surrogates
- **Encoding behaviour per mode**: URL encoding, Java encoding, custom
replacements
- **String translation**: ASCII passthrough, single/multiple surrogate pairs
- **Schemaless records**: String values, maps (all/filtered fields), arrays,
null handling
- **Schema-based records**: Structs with selective field translation, nested
structures
- **Key vs. Value transforms**: Correct record updates for each variant
- **Edge cases**: Unsupported types, multiple consecutive surrogate pairs
## Key improvements over earlier draft (PR #10287)
- Use `ConfigDef.LambdaValidator` instead of anonymous `Validator` classes
- `URLEncoder.encode(str, Charset)` instead of `(str, String)` — no checked
exception
- Fixed `Key.newRecord()` — original had key/value schemas swapped
- Null-value guard in `apply()` — original would NPE on null values
- `fields` config respected consistently for both MAP and STRUCT paths
JIRA: https://issues.apache.org/jira/browse/KAFKA-12443
Reviewers: Mickael Maison <[email protected]>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
