PDGGK opened a new pull request, #13181:
URL: https://github.com/apache/seatunnel/pull/13181

   ### Purpose of this pull request
   
   Fixes #9537, which the stale bot closed without a fix.
   
   With `array_mode = true`, `HttpSinkWriter.flush()` serializes each row to 
JSON and adds the text to the request array with `ArrayNode.add(String)`. That 
adds a string node, so the request body is an array of JSON strings rather than 
an array of objects:
   
   ```
   
["{\"id\":1,\"name\":\"xiaoming1\",\"age\":18}","{\"id\":2,\"name\":\"xiaoming2\",\"age\":19}","{\"id\":3,\"name\":\"xiaoming3\",\"age\":20}"]
   ```
   
   An endpoint that expects an array of objects rejects this body, as in the 
issue (`http response status code:[400]`). Object mode (`array_mode = false`) 
is not affected.
   
   This PR parses each serialized row with `mapper.readTree(...)` and adds it 
as a JSON value. Both serializers that `HttpSinkWriter` is used with in this 
repo produce JSON: the default `JsonSerializationSchema` and the WeChat sink's 
`WeChatBotMessageSerializationSchema`.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. With `array_mode = true`, the request body is now 
`[{"id":1,...},{"id":2,...}]`. The line has not changed since batch writes were 
added in #9292, so 2.3.11 through 3.0.0 and current `dev` send the array of 
strings.
   
   I sent the issue's three rows through `HttpSinkWriter` (with the real 
`HttpClientProvider` and `array_mode = true`) to a small HTTP server that 
accepts only a JSON array of objects. Both ran in Docker.
   
   | Jars | Body received | Result |
   |---|---|---|
   | 3.0.0 release | `["{\"id\":1,\"name\":\"xiaoming1\",\"age\":18}", ...]` | 
400. The writer logs the error and `close()` returns normally, so the batch is 
lost. |
   | `dev` (f54bb58) | same | 400. The write fails with 
`HttpConnectorException` (`HTTP-02`). |
   | `dev` + this PR | `[{"id":1,"name":"xiaoming1","age":18}, ...]` | 200 |
   
   ### How was this patch tested?
   
   - Added `HttpSinkBatchWriterTest#testArrayModeSendsEachRowAsJsonObject`. It 
writes three rows in array mode, parses the captured request body, and checks 
that it is an array of three objects with the expected `id`, `name` and `age`. 
The existing `testArrayModeWithBatch` only checks that the body starts with `[` 
and ends with `]`.
     - Without the change to `HttpSinkWriter`, it fails with 
`["{\"id\":1,\"name\":\"user1\",\"age\":20}", ...] ==> expected: <true> but 
was: <false>`.
     - With the change, the `connector-http-base` tests pass on JDK 17: 69 
tests, 0 failures.
   - `mvn spotless:check -pl 
seatunnel-connectors-v2/connector-http/connector-http-base` passes.
   
   #12376 also edits `HttpSinkBatchWriterTest`. The new test here is placed 
before `testArrayModeWithBatch`, so it does not touch the methods that PR 
changes.
   
   ### Check list
   
   * [ ] If any new Jar binary package adding in your PR, please add License 
Notice according
     [New License 
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
   * [ ] If necessary, please update the documentation to describe the new 
feature. https://github.com/apache/seatunnel/tree/dev/docs
   * [ ] If necessary, please update `incompatible-changes.md` to describe the 
incompatibility caused by this PR.
   * [ ] If you are contributing the connector code, please check that the 
following files are updated:
     1. Update 
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
 and add new connector information in it
     2. Update the pom file of 
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
     3. Add ci label in 
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
     4. Add e2e testcase in 
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
     5. Update connector 
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to