Rangsh opened a new pull request, #12172:
URL: https://github.com/apache/seatunnel/pull/12172
Skip unused envelope fields during deserialize and reuse source maps / UTF-8
byte writes on serialize to cut temporary object churn (#12074).
<!--
Thank you for contributing to SeaTunnel! Please make sure that your code
changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
## Contribution Checklist
- Make sure that the pull request corresponds to a [GITHUB
issue](https://github.com/apache/seatunnel/issues).
- Name the pull request in the form "[Feature] [component] Title of the
pull request", where *Feature* can be replaced by `Hotfix`, `Bug`, etc.
- Minor fixes should be named following this pattern: `[hotfix] [docs] Fix
typo in README.md doc`.
-->
### Purpose of this pull request
This PR addresses the Debezium JSON SerDe performance challenge in
[#12074](https://github.com/apache/seatunnel/issues/12074).
Profiler evidence in the issue showed that deserialization spends most of
its time in Jackson `ObjectMapper.readTree` / `ObjectNode` construction and the
related `HashMap` work, because the full envelope (including large unused
objects such as `source`, `transaction`, and optional top-level `schema`) was
materialized on every event. Serialization also allocated a fresh `source` map
and called `TablePath.of` repeatedly, and `JsonSerializationSchema` paid for an
intermediate `String` on the common UTF-8 path.
Changes:
- **Deserialize**: stream the envelope with `JsonParser` and materialize
only `op`, `ts_ms`, `before`, and `after` (under `payload` when schema is
included); skip unused objects. Cache `tableId` and share row metadata
application.
- **`DebeziumRowConverter`**: avoid double field lookup (`has` + `get` →
`get`), cache root field metadata, and use `asLong()` for numeric timestamps.
- **Serialize**: reuse the `source` map, cache parsed `TablePath` components
per `tableId`, read event time via `getOptionsOrNull()`, and use
`ObjectMapper.writeValueAsBytes` for UTF-8 in `JsonSerializationSchema`.
### Does this PR introduce _any_ user-facing change?
No. Behavior and Debezium JSON envelope shape are unchanged. This is an
internal performance optimization of the existing SerDe path.
### How was this patch tested?
- Existing unit tests:
- `DebeziumJsonSerDeSchemaTest`
- `DebeziumJsonDeserializationSchemaDispatcherTest`
- `JsonRowDataSerDeSchemaTest`
- Local JMH on Temurin 11 with `DebeziumJsonFormatBenchmark` and `-prof gc`.
Local after-change results (absolute throughput depends on the machine;
allocation `B/op` is the primary comparison to the issue baseline):
| Benchmark | Score (ops/ms) | `gc.alloc.rate.norm` (B/op) |
| --- | ---: | ---: |
| `deserializeInsertEvent` | 777.399 ± 123.719 | **3024.002** |
| `deserializeUpdateEvent` | 549.639 ± 119.749 | 4416.002 |
| `serializeInsertEvent` | 989.818 ± 27.224 | 2040.001 |
| `serializeMergedUpdateEvent` | 686.789 ± 25.287 | 3208.002 |
Issue [#12074](https://github.com/apache/seatunnel/issues/12074) Java 11
diagnostic baseline for `deserializeInsertEvent` allocation was **5176.492
B/op** (challenge target ≈ 4100 B/op). After this change, local allocation for
that method is **3024 B/op** (~41% lower than the published baseline).
Reproduce:
```bash
./mvnw -Pbenchmark -pl seatunnel-benchmarks -am -DskipTests package
java -jar seatunnel-benchmarks/target/benchmarks.jar
'DebeziumJsonFormatBenchmark' \
-f 1 -wi 3 -i 5 -w 5s -r 5s -tu ms -prof gc
```
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
— N/A, no new jars
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs — N/A, no
user-facing change
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR. — N/A, backward compatible
* [x] If you are contributing the connector code, please check that the
following files are updated: — N/A, Format module only
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
Fixes #12074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
