hansva commented on code in PR #9573:
URL: https://github.com/apache/hop/pull/9573#discussion_r4614721516


##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(
+      RestConnection conn, PaginationState state, RestExchangeResult ex, 
String resolvedSplitPath)
       throws HopException {
-    List<String> items =
-        Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body, 
resolvedSplitPath);
-    if (items != null) {
-      return !items.isEmpty();
+    String itemsExpression = pagingExpression(conn.getItemsJsonPath(), 
conn.getItemsXPath());
+    String countingExpression =
+        Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+    if (Utils.isEmpty(countingExpression)) {
+      return !Utils.isEmpty(NVL(ex.body, "").trim());
+    }
+    List<String> items = splitResultItems(ex.body, countingExpression);
+    if (items.isEmpty()) {
+      return false;
+    }
+    state.itemsRead += items.size();
+
+    Long totalCount = readTotalCount(conn, ex.body);
+    if (totalCount != null && state.itemsRead >= totalCount) {
+      if (isDetailed()) {
+        logDetailed(
+            BaseMessages.getString(
+                PKG, "Rest.Log.PaginationTotalCountReached", state.itemsRead, 
totalCount));
+      }
+      return false;
+    }
+    return true;
+  }
+
+  /**
+   * Picks the JsonPath or the XPath of a pair of paging expressions, the way 
the cursor and
+   * next-URL expressions are picked: the JsonPath when the response is read 
as JSON, otherwise the
+   * XPath for an XML response. A JsonPath with no XPath next to it is always 
used, so a mistake in
+   * it fails loudly rather than being ignored. Returns {@code null} when 
neither applies.
+   */
+  private String pagingExpression(String jsonPath, String xPath) {

Review Comment:
   **XPaths are silently dropped for non-XML application types.** When only an 
XPath is set and the application type is e.g. `TEXT PLAIN` or `FORM URLENCODED` 
(which people do use for XML APIs), this returns `null` without a log line. 
Paging then falls back to the split path or the empty-body rule, which never 
ends for an envelope. That goes against the javadoc's "fails loudly". Please 
log a warning (or throw) when an XPath is set but cannot be used.



##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -927,6 +930,7 @@ protected void runPaginationLoop(Object[] rowData) throws 
HopException {
         connection != null ? connection.getPaginationType() : 
RestPaginationType.NONE;
     PaginationState state = new PaginationState();
     state.effectiveLimit = resolvePagingLimit(connection);
+    state.pageNumber = resolveFirstPageNumber(connection);

Review Comment:
   **The first page number is resolved for every paging type.** The editor 
keeps the fields of the other paging types. So a connection that was once 
`PAGE_NUMBER` with first page `${START_PAGE}` (or `-1`), and was later switched 
to `OFFSET_LIMIT`, `CURSOR` or `LINK_HEADER`, fails with 
`Rest.Error.PaginationFirstPageNumber` when `START_PAGE` isn't set, although 
the field is ignored for that type. Please only call `resolveFirstPageNumber` 
when the type is `PAGE_NUMBER`.



##########
plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnectionEditor.java:
##########
@@ -948,10 +985,60 @@ private void buildPaginationWidgets(Composite p) {
     fdNx.left = new FormAttachment(middle, margin);
     fdNx.right = new FormAttachment(100, -margin);
     wNextPageUrlXPath.setLayoutData(fdNx);
+    lastControl = wNextPageUrlXPath;
+
+    wItemsJsonPath =
+        addPaginationTextRow(
+            p,
+            lastControl,
+            "RestConnectionEditor.Pagination.ItemsJson.Label",
+            "RestConnectionEditor.Pagination.ItemsJson.Tooltip");
+    wItemsXPath =
+        addPaginationTextRow(
+            p,
+            wItemsJsonPath,
+            "RestConnectionEditor.Pagination.ItemsXPath.Label",

Review Comment:
   **The XPath rows reuse the JsonPath tooltips.** This XPath row (and the 
Total count XPath row) uses the `...Json.Tooltip` key, which suggests `$.list` 
and `$.count`. Please add `ItemsXPath`/`TotalCountXPath` tooltip keys with 
XPath examples.



##########
plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnectionEditor.java:
##########
@@ -948,10 +985,60 @@ private void buildPaginationWidgets(Composite p) {
     fdNx.left = new FormAttachment(middle, margin);
     fdNx.right = new FormAttachment(100, -margin);
     wNextPageUrlXPath.setLayoutData(fdNx);
+    lastControl = wNextPageUrlXPath;
+
+    wItemsJsonPath =
+        addPaginationTextRow(
+            p,
+            lastControl,
+            "RestConnectionEditor.Pagination.ItemsJson.Label",
+            "RestConnectionEditor.Pagination.ItemsJson.Tooltip");
+    wItemsXPath =
+        addPaginationTextRow(
+            p,
+            wItemsJsonPath,
+            "RestConnectionEditor.Pagination.ItemsXPath.Label",
+            "RestConnectionEditor.Pagination.ItemsJson.Tooltip");
+    wTotalCountJsonPath =
+        addPaginationTextRow(
+            p,
+            wItemsXPath,
+            "RestConnectionEditor.Pagination.TotalCountJson.Label",
+            "RestConnectionEditor.Pagination.TotalCountJson.Tooltip");
+    wTotalCountXPath =
+        addPaginationTextRow(
+            p,
+            wTotalCountJsonPath,
+            "RestConnectionEditor.Pagination.TotalCountXPath.Label",
+            "RestConnectionEditor.Pagination.TotalCountJson.Tooltip");
 
     refreshPaginationSensitiveFields();
   }
 
+  /** One label and text field below {@code above} on the Pagination tab. */
+  private TextVar addPaginationTextRow(

Review Comment:
   **Six more hand-laid FormAttachment rows.** `addPaginationTextRow` adds six 
of them, plus a hand-built ScrolledComposite with a min-height listener and 
manual copying in `setWidgetsContent`/`getWidgetsContent`. The new 
`RestConnection` fields have no `@GuiWidgetElement`. Our dialog convention is 
annotated fields in a grouped `GuiCompositeWidgets` layout (tabs/boxes), which 
also scrolls and resizes on its own. The editor was hand-built before this PR, 
so I'm fine with doing this in a follow-up, but I'd rather not grow the 
hand-built part further. At least the new pagination fields could be annotated.



##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(
+      RestConnection conn, PaginationState state, RestExchangeResult ex, 
String resolvedSplitPath)
       throws HopException {
-    List<String> items =
-        Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body, 
resolvedSplitPath);
-    if (items != null) {
-      return !items.isEmpty();
+    String itemsExpression = pagingExpression(conn.getItemsJsonPath(), 
conn.getItemsXPath());
+    String countingExpression =
+        Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+    if (Utils.isEmpty(countingExpression)) {
+      return !Utils.isEmpty(NVL(ex.body, "").trim());
+    }
+    List<String> items = splitResultItems(ex.body, countingExpression);

Review Comment:
   **An items XPath that points at the container never sees an empty page.** 
With the XPath counterpart of the documented `$.list` example, e.g. 
`/envelope/list`, the page past the end is 
`<envelope><count>6</count><list/></envelope>`. `splitResultItems` returns one 
node, which serialises to `<list/>`, not blank, so the page counts as 1 item 
and paging runs to `maxPagesLoops`. So the endless paging of #9549 is still 
there for XML. `itemsRead` also grows by 1 per page instead of by the item 
count, which breaks the total-count stop. JSON `$.list` works only because 
JsonPath returns an `Iterable`. For XPath, count the container's element 
children (or document that the path must select the items, e.g. 
`/envelope/list/item`).



##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(
+      RestConnection conn, PaginationState state, RestExchangeResult ex, 
String resolvedSplitPath)
       throws HopException {
-    List<String> items =
-        Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body, 
resolvedSplitPath);
-    if (items != null) {
-      return !items.isEmpty();
+    String itemsExpression = pagingExpression(conn.getItemsJsonPath(), 
conn.getItemsXPath());
+    String countingExpression =
+        Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+    if (Utils.isEmpty(countingExpression)) {
+      return !Utils.isEmpty(NVL(ex.body, "").trim());
+    }
+    List<String> items = splitResultItems(ex.body, countingExpression);
+    if (items.isEmpty()) {
+      return false;
+    }
+    state.itemsRead += items.size();
+
+    Long totalCount = readTotalCount(conn, ex.body);
+    if (totalCount != null && state.itemsRead >= totalCount) {
+      if (isDetailed()) {
+        logDetailed(
+            BaseMessages.getString(
+                PKG, "Rest.Log.PaginationTotalCountReached", state.itemsRead, 
totalCount));
+      }
+      return false;
+    }
+    return true;
+  }
+
+  /**
+   * Picks the JsonPath or the XPath of a pair of paging expressions, the way 
the cursor and
+   * next-URL expressions are picked: the JsonPath when the response is read 
as JSON, otherwise the
+   * XPath for an XML response. A JsonPath with no XPath next to it is always 
used, so a mistake in
+   * it fails loudly rather than being ignored. Returns {@code null} when 
neither applies.
+   */
+  private String pagingExpression(String jsonPath, String xPath) {
+    String appType = NVL(resolve(meta.getApplicationType()), "");
+    String json = resolveSplitPathOrPagingExpression(jsonPath);
+    String xml = resolveSplitPathOrPagingExpression(xPath);
+    if (!Utils.isEmpty(json) && (shouldParseResponseAsJson(appType, json) || 
Utils.isEmpty(xml))) {
+      return json;
+    }
+    if (!Utils.isEmpty(xml) && isXmlApplicationType(appType)) {
+      return xml;
+    }
+    return null;
+  }
+
+  private static boolean isXmlApplicationType(String appType) {

Review Comment:
   **The XML type list now exists twice.** `isXmlApplicationType()` copies the 
five-type check that `splitResultItems()` still spells out inline. If a type is 
added to only one of them, `pagingExpression()` picks the XPath while 
`splitResultItems()` throws, or the other way round. Please have 
`splitResultItems()` call this method.



##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(
+      RestConnection conn, PaginationState state, RestExchangeResult ex, 
String resolvedSplitPath)
       throws HopException {
-    List<String> items =
-        Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body, 
resolvedSplitPath);
-    if (items != null) {
-      return !items.isEmpty();
+    String itemsExpression = pagingExpression(conn.getItemsJsonPath(), 
conn.getItemsXPath());
+    String countingExpression =
+        Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+    if (Utils.isEmpty(countingExpression)) {
+      return !Utils.isEmpty(NVL(ex.body, "").trim());
+    }
+    List<String> items = splitResultItems(ex.body, countingExpression);
+    if (items.isEmpty()) {
+      return false;
+    }
+    state.itemsRead += items.size();
+
+    Long totalCount = readTotalCount(conn, ex.body);
+    if (totalCount != null && state.itemsRead >= totalCount) {
+      if (isDetailed()) {
+        logDetailed(
+            BaseMessages.getString(
+                PKG, "Rest.Log.PaginationTotalCountReached", state.itemsRead, 
totalCount));
+      }
+      return false;
+    }
+    return true;
+  }
+
+  /**
+   * Picks the JsonPath or the XPath of a pair of paging expressions, the way 
the cursor and
+   * next-URL expressions are picked: the JsonPath when the response is read 
as JSON, otherwise the
+   * XPath for an XML response. A JsonPath with no XPath next to it is always 
used, so a mistake in
+   * it fails loudly rather than being ignored. Returns {@code null} when 
neither applies.
+   */
+  private String pagingExpression(String jsonPath, String xPath) {
+    String appType = NVL(resolve(meta.getApplicationType()), "");
+    String json = resolveSplitPathOrPagingExpression(jsonPath);
+    String xml = resolveSplitPathOrPagingExpression(xPath);
+    if (!Utils.isEmpty(json) && (shouldParseResponseAsJson(appType, json) || 
Utils.isEmpty(xml))) {
+      return json;
+    }
+    if (!Utils.isEmpty(xml) && isXmlApplicationType(appType)) {
+      return xml;
+    }
+    return null;
+  }
+
+  private static boolean isXmlApplicationType(String appType) {
+    return RestMeta.APPLICATION_TYPE_XML.equals(appType)
+        || RestMeta.APPLICATION_TYPE_TEXT_XML.equals(appType)
+        || RestMeta.APPLICATION_TYPE_ATOM_XML.equals(appType)
+        || RestMeta.APPLICATION_TYPE_SVG_XML.equals(appType)
+        || RestMeta.APPLICATION_TYPE_XHTML.equals(appType);
+  }
+
+  /**
+   * The total number of items the API reports through the connection's total 
count path, or {@code
+   * null} when no path is set or the page does not hold a whole number there.
+   */
+  private Long readTotalCount(RestConnection conn, String body) throws 
HopException {
+    String expression = pagingExpression(conn.getTotalCountJsonPath(), 
conn.getTotalCountXPath());
+    if (Utils.isEmpty(expression)) {
+      return null;
+    }
+    String value;
+    try {
+      // The same JSON-or-XML rule as splitResultItems(), so the count and the 
items agree.
+      if (shouldParseResponseAsJson(NVL(resolve(meta.getApplicationType()), 
""), expression)) {
+        Object raw =
+            JsonPath.using(JSON_PATH_CONFIGURATION).parse(NVL(body, 
"{}")).read(expression);
+        if (raw instanceof Number number) {
+          return number.longValue();
+        }
+        value = raw == null ? null : raw.toString();
+      } else {
+        Document doc;
+        try (StringReader sr = new StringReader(NVL(body, ""))) {
+          doc = documentBuilderFactory().newDocumentBuilder().parse(new 
InputSource(sr));

Review Comment:
   **A body that can't be parsed fails the whole transform.** A `SAXException` 
here, or a JSON body that JsonPath can't parse (`SUPPRESS_EXCEPTIONS` doesn't 
cover `parse()`), becomes a fatal `Rest.Error.PaginationTotalCount`. A missing 
or non-numeric count a few lines further down only logs and continues. I'd 
treat a parse failure the same way: log it and continue as if there were no 
count.



##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(
+      RestConnection conn, PaginationState state, RestExchangeResult ex, 
String resolvedSplitPath)
       throws HopException {
-    List<String> items =
-        Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body, 
resolvedSplitPath);
-    if (items != null) {
-      return !items.isEmpty();
+    String itemsExpression = pagingExpression(conn.getItemsJsonPath(), 
conn.getItemsXPath());

Review Comment:
   **The items path and the split path can disagree.** The items path decides 
whether a page is empty, but `emitPagedResultRows` still emits rows by the 
split path. With items `$.list` and split `$.data[*]` (a mistake or an API 
change), paging continues while every page emits 0 rows, and at the end 
`enforcePagingSplitEmittedSomething` throws `PagingSplitProducedZeroRows`, 
pointing at the split path rather than the mismatch. In the reverse case, 
paging stops after the first page while the split path would have found rows. 
Could the stop rule use the split path's result when both are set, or at least 
report the mismatch clearly?



##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(
+      RestConnection conn, PaginationState state, RestExchangeResult ex, 
String resolvedSplitPath)
       throws HopException {
-    List<String> items =
-        Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body, 
resolvedSplitPath);
-    if (items != null) {
-      return !items.isEmpty();
+    String itemsExpression = pagingExpression(conn.getItemsJsonPath(), 
conn.getItemsXPath());
+    String countingExpression =
+        Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+    if (Utils.isEmpty(countingExpression)) {

Review Comment:
   **The total count path is ignored here.** With a Total count path 
(`$.count`) but no items path and no result split path, `countingExpression` is 
empty and we return on the non-empty-body check before `readTotalCount()` runs. 
An envelope API never sends an empty body, so paging runs to `maxPagesLoops` 
and emits duplicate envelope rows, and the user gets no hint that the setting 
had no effect. Either apply the total count here too, or log a warning (or fail 
validation) that a total count needs an items path.



##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(

Review Comment:
   **Each page body is parsed several times.** It is parsed up to 4 times per 
page: the split in `emitPagedResultRows`, the items path, the total count and 
the cursor. On top of that, `pagingExpression()` builds 
`substitutionMapSnapshot()` twice per call, twice per page. For pages with 
thousands of items that's wasted CPU and memory in the paging loop. Suggest 
resolving the items and count expressions once in `runPaginationLoop`, and 
parsing the body once per page.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to