bamaer commented on code in PR #9573:
URL: https://github.com/apache/hop/pull/9573#discussion_r4625258471


##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(
+      RestConnection conn, PaginationState state, RestExchangeResult ex, 
String resolvedSplitPath)
       throws HopException {
-    List<String> items =
-        Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body, 
resolvedSplitPath);
-    if (items != null) {
-      return !items.isEmpty();
+    String itemsExpression = pagingExpression(conn.getItemsJsonPath(), 
conn.getItemsXPath());

Review Comment:
   The items path keeps deciding the stop: a split like `$.page` (one row per 
page) legitimately finds something on the empty page too. Both directions of a 
mismatch are now logged once, with both paths: items found but no rows from the 
split path, and no items on the first page while the split path gave rows. 
Tests: `itemsWithoutRowsAreLoggedOnce`, 
`rowsWithoutItemsOnTheFirstPageAreLogged`.



##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(

Review Comment:
   Each page body is now parsed once, as JSON or XML, and shared by the split, 
items, total count, cursor and next-URL paths. The items and total count 
expressions are resolved once per input row.



##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(
+      RestConnection conn, PaginationState state, RestExchangeResult ex, 
String resolvedSplitPath)
       throws HopException {
-    List<String> items =
-        Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body, 
resolvedSplitPath);
-    if (items != null) {
-      return !items.isEmpty();
+    String itemsExpression = pagingExpression(conn.getItemsJsonPath(), 
conn.getItemsXPath());
+    String countingExpression =
+        Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+    if (Utils.isEmpty(countingExpression)) {
+      return !Utils.isEmpty(NVL(ex.body, "").trim());
+    }
+    List<String> items = splitResultItems(ex.body, countingExpression);
+    if (items.isEmpty()) {
+      return false;
+    }
+    state.itemsRead += items.size();
+
+    Long totalCount = readTotalCount(conn, ex.body);
+    if (totalCount != null && state.itemsRead >= totalCount) {
+      if (isDetailed()) {
+        logDetailed(
+            BaseMessages.getString(
+                PKG, "Rest.Log.PaginationTotalCountReached", state.itemsRead, 
totalCount));
+      }
+      return false;
+    }
+    return true;
+  }
+
+  /**
+   * Picks the JsonPath or the XPath of a pair of paging expressions, the way 
the cursor and
+   * next-URL expressions are picked: the JsonPath when the response is read 
as JSON, otherwise the
+   * XPath for an XML response. A JsonPath with no XPath next to it is always 
used, so a mistake in
+   * it fails loudly rather than being ignored. Returns {@code null} when 
neither applies.
+   */
+  private String pagingExpression(String jsonPath, String xPath) {
+    String appType = NVL(resolve(meta.getApplicationType()), "");
+    String json = resolveSplitPathOrPagingExpression(jsonPath);
+    String xml = resolveSplitPathOrPagingExpression(xPath);
+    if (!Utils.isEmpty(json) && (shouldParseResponseAsJson(appType, json) || 
Utils.isEmpty(xml))) {
+      return json;
+    }
+    if (!Utils.isEmpty(xml) && isXmlApplicationType(appType)) {
+      return xml;
+    }
+    return null;
+  }
+
+  private static boolean isXmlApplicationType(String appType) {

Review Comment:
   Done, `splitResultItems()` uses `isXmlApplicationType()` now.



##########
plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnectionEditor.java:
##########
@@ -948,10 +985,60 @@ private void buildPaginationWidgets(Composite p) {
     fdNx.left = new FormAttachment(middle, margin);
     fdNx.right = new FormAttachment(100, -margin);
     wNextPageUrlXPath.setLayoutData(fdNx);
+    lastControl = wNextPageUrlXPath;
+
+    wItemsJsonPath =
+        addPaginationTextRow(
+            p,
+            lastControl,
+            "RestConnectionEditor.Pagination.ItemsJson.Label",
+            "RestConnectionEditor.Pagination.ItemsJson.Tooltip");
+    wItemsXPath =
+        addPaginationTextRow(
+            p,
+            wItemsJsonPath,
+            "RestConnectionEditor.Pagination.ItemsXPath.Label",

Review Comment:
   Added `ItemsXPath` / `TotalCountXPath` tooltips with XPath examples.



##########
plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnectionEditor.java:
##########
@@ -948,10 +985,60 @@ private void buildPaginationWidgets(Composite p) {
     fdNx.left = new FormAttachment(middle, margin);
     fdNx.right = new FormAttachment(100, -margin);
     wNextPageUrlXPath.setLayoutData(fdNx);
+    lastControl = wNextPageUrlXPath;
+
+    wItemsJsonPath =
+        addPaginationTextRow(
+            p,
+            lastControl,
+            "RestConnectionEditor.Pagination.ItemsJson.Label",
+            "RestConnectionEditor.Pagination.ItemsJson.Tooltip");
+    wItemsXPath =
+        addPaginationTextRow(
+            p,
+            wItemsJsonPath,
+            "RestConnectionEditor.Pagination.ItemsXPath.Label",
+            "RestConnectionEditor.Pagination.ItemsJson.Tooltip");
+    wTotalCountJsonPath =
+        addPaginationTextRow(
+            p,
+            wItemsXPath,
+            "RestConnectionEditor.Pagination.TotalCountJson.Label",
+            "RestConnectionEditor.Pagination.TotalCountJson.Tooltip");
+    wTotalCountXPath =
+        addPaginationTextRow(
+            p,
+            wTotalCountJsonPath,
+            "RestConnectionEditor.Pagination.TotalCountXPath.Label",
+            "RestConnectionEditor.Pagination.TotalCountJson.Tooltip");
 
     refreshPaginationSensitiveFields();
   }
 
+  /** One label and text field below {@code above} on the Pagination tab. */
+  private TextVar addPaginationTextRow(

Review Comment:
   Agreed. Rather than add annotated fields to the hand-built tab, I will 
convert the whole editor to annotated widgets in a follow-up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to