Hisoka-X commented on code in PR #9103:
URL: https://github.com/apache/seatunnel/pull/9103#discussion_r2043417633
##########
seatunnel-connectors-v2/connector-http/connector-http-base/pom.xml:
##########
@@ -69,5 +71,35 @@
<artifactId>json-path</artifactId>
<version>${json-path.version}</version>
</dependency>
+
+ <!-- Test dependencies -->
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-api</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-engine</artifactId>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-core</artifactId>
+ <version>${mockito.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.mockito</groupId>
+ <artifactId>mockito-junit-jupiter</artifactId>
+ <version>${mockito.version}</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>cn.hutool</groupId>
+ <artifactId>hutool-json</artifactId>
+ <version>${hutool-json.version}</version>
+ <scope>test</scope>
Review Comment:
Please do not introduce new json dependency. Just use jackson enough.
##########
seatunnel-connectors-v2/connector-http/connector-http-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceReader.java:
##########
@@ -257,27 +257,139 @@ private List<Map<String, String>>
parseToMap(List<List<String>> datas, JsonField
private List<List<String>> decodeJSON(String data) {
ReadContext jsonReadContext =
JsonPath.using(jsonConfiguration).parse(data);
- List<List<String>> results = new ArrayList<>(jsonPaths.length);
- for (JsonPath path : jsonPaths) {
- List<String> result = jsonReadContext.read(path);
- results.add(result);
+
+ if (httpParameter.isJsonFiledMissedReturnNull()) {
+ // Extract the common parent path from all configured paths
+ String commonParentPath = extractCommonParentPath(jsonPaths);
+ if (commonParentPath == null) {
+ throw new HttpConnectorException(
+ HttpConnectorErrorCode.FIELD_DATA_IS_INCONSISTENT,
+ "Could not find common parent path in JsonPaths. All
paths must share a common array parent.");
+ }
+
+ // Get all objects under the common parent path
+ List<Map<String, Object>> objects;
+ try {
+ objects = jsonReadContext.read(commonParentPath);
+ } catch (Exception e) {
Review Comment:
The current implementation is a bit complicated.
In fact, the jsonpath can do this by itself. In jsonpath,
https://github.com/apache/seatunnel/blob/d2ac555db1772cf58342db7709015a3b594aa87e/seatunnel-connectors-v2/connector-http/connector-http-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceReader.java#L64
we already enabled `DEFAULT_PATH_LEAF_TO_NULL`,
according the comment, when some path leaf not existed, it should return
null. That's what we want, so we should check why this configure not work first.
##########
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/java/org/apache/seatunnel/e2e/connector/http/HttpIT.java:
##########
@@ -357,6 +357,9 @@ public void testSourceToAssertSink(TestContainer container)
Container.ExecResult execResult19 =
container.executeJob("/http_page_increase_start_num.conf");
Assertions.assertEquals(0, execResult19.getExitCode());
+
+ Container.ExecResult execResult21 =
container.executeJob("/http_to_console.conf");
+ Assertions.assertEquals(0, execResult21.getExitCode());
Review Comment:
Let's remove this e2e test case @ocean-zhc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
