SEPURI-SAI-KRISHNA opened a new pull request, #13222:
URL: https://github.com/apache/seatunnel/pull/13222
Yes, glad to contribute to Apache SeaTunnel.
### Purpose of this pull request
Closes #13220.
GraphQL servers report a rejected query or mutation with HTTP 200 and an
`errors` array. The connector never looked at `errors`, so rejected requests
passed as successes. This PR fixes that and the two problems that appear once
errors are no longer hidden:
- **Errors fail the request.** A new `GraphQLUtil.checkNoErrors` throws when
the response has a non-empty `errors` array, with the server's errors in the
message. `GraphQLSinkWriter.write` calls it (`GraphQL-03`), and also throws
`HTTP-02` on a status other than 200, where it used to only log and return.
`GraphQLSourceHttpReader` calls it before each `collect`, in both the
single-body and the line-by-line branch (`GraphQL-01`). A response that is not
a JSON object, or has an empty `errors` array, is left to the existing parsing.
- **`variables` not set.** `GraphQLSourceParameter` and
`GraphQLSinkParameter` put the string `"{}"` into the body, so servers got
`"variables":"{}"`. Hasura rejects that, and the sink failed at its first row
with `ClassCastException` when it cast the value to a `Map`. They now put an
empty map.
- **Date and time fields.** The sink builds the body with Gson, which has no
`java.time` adapter: on Java 11 a `LocalDateTime` became a nested object, and
on Java 17 Gson threw `JsonIOException`. `GraphQLSinkWriter.toVariableValue`
now sends `LocalDateTime`, `LocalDate`, `LocalTime` and `OffsetDateTime` as ISO
8601 strings. Other values are unchanged.
WebSocket subscription messages are not checked; that path is unchanged.
**E2E.** `GraphQLIT` passed before because every response was an error that
nothing checked. Its tables were never tracked in Hasura, so in the #12273 run
all 21 `/v1/graphql` responses were HTTP 200 with errors (`field 'source' not
found`, `no mutations exist`). Now:
- `startUp` tracks `source` and `sink` through Hasura's metadata API.
- `testGraphQLSourceAndSink` asserts the exit code with stderr as the
message, empties `sink`, runs the sink job and checks both rows in PostgreSQL.
- `graphql_to_assert.conf` requires exactly 2 rows.
- `fake_to_graphql.conf` declares `$val_double: float8!`. Hasura rejected
`Float!` for a `double precision` column, which nobody saw until now.
- New `testRejectedMutationFailsTheJob` and `testRejectedQueryFailsTheJob`
use a table that does not exist and assert a failed job with the server's
message. The query job sets no `variables`, so it also covers the empty-object
fix.
- Both rejected job tests look for the server's message in the job output
and in the engine logs, as `RedisMasterAndSlaveIT` does. On Flink the client
prints one failure cause, and when the failed task also loses an operator event
(`NoMoreElementEvent`), that is reported instead of the GraphQL error.
- The rejected query job sets `execution.restart.strategy = "no"`. The Flink
1.13, 1.15 and 1.18 images restart a failed job twice, and on the second
restart `SingleSplitEnumerator.addSplitsBack` throws
`IndexOutOfBoundsException`, which replaces the GraphQL error in the job
result. That is a separate bug in `connector-common`, and I will raise it
separately.
- Both rejected jobs set `job.retry.times = 0`, as the Mailchimp and Doris
E2Es do. The rejection fails every attempt, so a Zeta restore only delays the
result, and in one fork run the third restore of the rejected query job hit a
separate Zeta bug (a final checkpoint barrier from the failed attempt reached
the restored tasks) and the job stayed `RUNNING` until CI cancelled it after 5
hours. That bug is #13208, with a fix in #13209.
### Does this PR introduce _any_ user-facing change?
Yes. A job whose requests the server rejects now fails instead of finishing
with missing rows. Date and time values reach the server as ISO strings.
Documented in the GraphQL source and sink pages and in
`incompatible-changes.md`, EN and ZH. No option or default changes.
### How was this patch tested?
- 10 new unit tests (`GraphQLSinkWriterTest` 6,
`GraphQLSourceHttpReaderTest` 4) against a local JDK `HttpServer`: errors in a
200 response, partial data with errors, a 503, an empty `errors` array, the ISO
strings, and the body sent without `variables`. 11 of 11 module tests pass on
JDK 17, and the earlier 9 also with the test JVM on JDK 11. `spotless:check`
passes.
- With each fix reverted in turn, its tests fail: no errors check in the
sink, the non-200 log only, raw date values, no check in the source, and the
`"{}"` default (`{"variables":"{}",...}` and `ClassCastException`).
- `GraphQLIT` on Zeta locally: 3 of 3 pass. The source job reads 2 rows, the
sink job writes 2 rows and the test finds both in PostgreSQL. The rejected
query fails with `field 'missing_table' not found in type: 'query_root'`, and
the rejected mutation with `field 'insert_missing_table' not found in type:
'mutation_root'`. An earlier local run showed the `float8` problem, and one
before the `variables` fix showed `parsing HashMap failed` for the query job.
Flink and Spark were not run locally; the fork run covers them.
- The docs changes do not conflict with #11772, #12409 or #13216; each
merges with this branch without conflicts.
Fork run:
https://github.com/SEPURI-SAI-KRISHNA/seatunnel/actions/runs/42746905090 passes
on head `531886b183` (dev merged).
### Check list
* [X] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
(no new jar)
* [X] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs: GraphQL source and
sink pages, EN and ZH.
* [X] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR: added, EN and ZH.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]