alexey-bass opened a new pull request, #13276:
URL: https://github.com/apache/seatunnel/pull/13276
### Purpose of this pull request
Fixes #13273.
`JobInfoService.decodeJobBasicInfo` (used by `GET /running-jobs/summary`)
read `jobName` and `createTime` straight from the serialized
`JobImmutableInformation`, but did not skip the one-byte header that
Hazelcast's `DataSerializableSerializer` writes before the factory id and class
id. Every read was shifted by one byte, so the string length came out as
`(jobId & 0xFF) << 24`. For normal jobs that read hit EOF and the
full-deserialization fallback hid the problem; once the serialized information
was larger than that length, the read succeeded and the summary returned
megabytes of payload as `jobName`, with a wrong `createTime`.
The change is limited to the summary decoder:
- The fast path now follows the serializer's read side: checks the `Data`
type id, reads the header byte, requires the identified flag, checks the
factory id and class id against `JobDataSerializerHook`, and skips the two
version bytes when the header has the versioned flag.
- The decoded values are validated before use: the job id must equal the map
key, and the job name length is read and checked (not negative beyond the null
marker, at most 64 KiB, not more than the remaining bytes) before the string
itself is read.
- If any check fails or a read throws, the existing full deserialization is
used, as before.
`/running-jobs`, job state and the persisted `JobImmutableInformation`
format are not touched.
### Does this PR introduce _any_ user-facing change?
Yes, a bug fix: `/running-jobs/summary` returns the real job name and create
time for jobs with a large `JobImmutableInformation` (the bug is present in
3.0.0 and on `dev`). Nothing changes for other jobs, apart from the fast path
now actually being used instead of the fallback.
Deterministic before/after from the new regression test (a real
`JobImmutableInformation` of about 20 MB, job id ending in `0x01`, one running
job):
| | summary JSON length | `jobName` |
|---|---|---|
| before (`dev`) | 100,663,082 characters | 16,777,216 bytes of the payload |
| after | 117 characters | `cdc_example` |
### How was this patch tested?
Added to `JobInfoServiceNullSafetyTest`:
- `shouldDecodeSameBasicInfoInFastPathAsFullDeserialization`: serializes
real `JobImmutableInformation` objects with `SerializationService.toData`
(several job ids, plain, empty, non-ASCII and null names) and asserts that the
fast path, invoked directly so the fallback cannot mask it, returns the same
`jobName` and `createTime` as `toObject(...)`.
- `shouldReturnBoundedSummaryForLargeJobImmutableInformation`: the
regression. Builds a serialized information larger than 16 MB with a job id
ending in `0x01`, calls `getRunningJobsSummaryJson()` and asserts the real
name, the real timestamp and a bounded response. On `dev` it fails with
`summary length was 100663082`.
- `shouldFallBackToFullDeserializationWhenFastPathRejectsData`: a name above
the fast-path limit, a job id that differs from the map key, and
non-DataSerializable data are rejected by the fast path; the first two are
still decoded correctly through the full deserialization.
The existing `shouldDecodeLegacyJobImmutableInformationInFastPath` is kept
(only its reflective call follows the new method signature).
`./mvnw -pl seatunnel-engine/seatunnel-engine-server test
-Dtest=JobInfoServiceNullSafetyTest`: 10 tests, all pass; `spotless:apply` run.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [ ] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]