ocean-zhc commented on code in PR #9103:
URL: https://github.com/apache/seatunnel/pull/9103#discussion_r2044199060
##########
seatunnel-connectors-v2/connector-http/connector-http-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceReader.java:
##########
@@ -257,27 +257,139 @@ private List<Map<String, String>>
parseToMap(List<List<String>> datas, JsonField
private List<List<String>> decodeJSON(String data) {
ReadContext jsonReadContext =
JsonPath.using(jsonConfiguration).parse(data);
- List<List<String>> results = new ArrayList<>(jsonPaths.length);
- for (JsonPath path : jsonPaths) {
- List<String> result = jsonReadContext.read(path);
- results.add(result);
+
+ if (httpParameter.isJsonFiledMissedReturnNull()) {
+ // Extract the common parent path from all configured paths
+ String commonParentPath = extractCommonParentPath(jsonPaths);
+ if (commonParentPath == null) {
+ throw new HttpConnectorException(
+ HttpConnectorErrorCode.FIELD_DATA_IS_INCONSISTENT,
+ "Could not find common parent path in JsonPaths. All
paths must share a common array parent.");
+ }
+
+ // Get all objects under the common parent path
+ List<Map<String, Object>> objects;
+ try {
+ objects = jsonReadContext.read(commonParentPath);
+ } catch (Exception e) {
Review Comment:
> The current implementation is a bit complicated. In fact, the jsonpath can
do this by itself. In jsonpath,
>
>
https://github.com/apache/seatunnel/blob/d2ac555db1772cf58342db7709015a3b594aa87e/seatunnel-connectors-v2/connector-http/connector-http-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceReader.java#L64
>
> we already enabled `DEFAULT_PATH_LEAF_TO_NULL`,
>
> according the comment, when some path leaf not existed, it should return
null. That's what we want, so we should check why this configure not work first.
DEFAULT_PATH_LEAF_TO_NULL can't be handled for complex, this configuration
is only effective for leaf nodes, and can't be handled for missing parent
nodes, see details:
<img width="1301" alt="image"
src="https://github.com/user-attachments/assets/b88d93c1-aaa2-49de-bbb1-feeebae33cf5";
/>
<img width="1655" alt="image"
src="https://github.com/user-attachments/assets/73c85e6f-bd77-4f96-bbaf-5ef27d82268e";
/>
Get parent node according to user configuration, e.g.
key1_1 = "$.result.rows[*].key1.key1_1"
key2_1 = "$.result.rows[*].key2.key2_1"
will get result.rows[*] as the parent node, and the child node is missing to
fill null
result：
<img width="837" alt="image"
src="https://github.com/user-attachments/assets/33254679-9b78-49ed-8efc-bb2dc9cde62c";
/>
unit test：
https://github.com/ocean-zhc/seatunnel/blob/dev/seatunnel-connectors-v2/connector-http/connector-http-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/http/JsonFieldMissedReturnNullComplexTest.java
https://github.com/ocean-zhc/seatunnel/blob/dev/seatunnel-connectors-v2/connector-http/connector-http-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/http/JsonFieldMissedReturnNullTest.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
