hutiefang76 opened a new issue, #12961:
URL: https://github.com/apache/seatunnel/issues/12961
### Search before asking
- [x] I searched open and closed issues/PRs and found no report of this
container-shape problem. #12871 concerns decimal precision and #12251 leaves
shared JSON converters unchanged.
### What happened
JSON fields declared as `ARRAY` or `MAP` accept incompatible JSON shapes
without a parse error. For example, an `ARRAY<INT>` field containing `"abc"`,
`42` or `true` becomes an empty array. An object such as `{"k":1}` becomes an
array containing null. A `MAP<STRING,INT>` field containing a string, number,
boolean or array becomes an empty map.
This happens even with `ignore-parse-errors=false`. With the option set to
true, these invalid fields also become empty collections instead of null. That
hides invalid source data as valid empty data. `JsonFormatOptions` already
describes the intended policy: errors fail by default, and fields are set to
null when parse errors are ignored.
I reproduced this through the public
`JsonDeserializationSchema.deserialize(byte[])` entry point on unmodified
production sources. The existing test class now has 31 tests: four fail,
covering 16 invalid input/mode combinations; valid arrays/maps, null
elements/values, empty collections, numeric-string coercion and byte/String
entry-point controls pass. No production code had been changed for that run.
I am working on a narrow follow-up to validate ARRAY/MAP node shapes before
conversion, using the existing parse-error policy. Positional-array ROW
conversion, valid values and configuration names/defaults will remain
unchanged. Patched verification is not complete yet.
### SeaTunnel Version
Current dev `6b63aca173eb8c056db0b78a87539aec3e4ad1a5`.
### SeaTunnel Config
A row with `id INT` and `arr ARRAY<INT>` reproduces the issue with
`{"id":7,"arr":"abc"}`. For `id INT` and `m MAP<STRING,INT>`, use
`{"id":7,"m":[1,2]}`. The mismatch is in the JSON format conversion itself; no
external connector service is needed for the reproduction.
Minimal ARRAY example:
```java
SeaTunnelRowType type = new SeaTunnelRowType(
new String[] {"id", "arr"},
new SeaTunnelDataType<?>[] {BasicType.INT_TYPE,
ArrayType.INT_ARRAY_TYPE});
JsonDeserializationSchema schema = new JsonDeserializationSchema(false,
false, type);
SeaTunnelRow row = schema.deserialize(
"{\"id\":7,\"arr\":\"abc\"}".getBytes(StandardCharsets.UTF_8));
// row.getField(1) is Integer[0]; strict mode did not reject the mismatch.
```
### Running Command
The regression was run on the unmodified converters with:
```shell
./mvnw -B -pl seatunnel-formats/seatunnel-format-json -am
-Dtest=JsonRowDataSerDeSchemaTest -Dsurefire.failIfNoSpecifiedTests=false
-DfailIfNoTests=false package
```
The four new tests fail their strict-mode exception and lenient-mode null
assertions; 31 tests total, 4 failures, 0 errors, 0 skipped.
### Error Exception
There is no exception for these invalid shapes. The regression reports
`Expected SeaTunnelRuntimeException to be thrown, but nothing was thrown` in
strict mode, and `expected: <null> but was: <[]>`, `<[null]>` or `<{}>` in
lenient mode.
### Zeta or Flink or Spark Version
Reproduced directly in the JSON format module; an engine is not required for
this conversion path.
### Java or Scala Version
JDK 17.0.18, macOS ARM64.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
Investigation and regression preparation are AI-assisted with Codex and
DeepSeek.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
