hansva commented on code in PR #9573:
URL: https://github.com/apache/hop/pull/9573#discussion_r4614721516
##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
}
/**
- * @return whether this response appears to carry at least one item (whole
body when no
- * split-path, or extracted elements when splitting).
+ * Whether page, offset and cursor paging should ask for another page after
this one.
+ *
+ * <p>A page is empty when the connection's items path finds nothing in it.
Without an items path
+ * the result split path decides, and without either an empty body does —
which an API that wraps
+ * its results in an envelope never sends (#9549). The items path wins over
the split path because
+ * it describes the API, while the split path describes the rows this
transform wants.
+ *
+ * <p>With a total count path as well, paging also stops once that many
items were read, so the
+ * last, empty, page is not fetched at all.
*/
- private boolean exchangeHadPayload(RestExchangeResult ex, String
resolvedSplitPath)
+ private boolean pageHadItemsAndMoreRemain(
+ RestConnection conn, PaginationState state, RestExchangeResult ex,
String resolvedSplitPath)
throws HopException {
- List<String> items =
- Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body,
resolvedSplitPath);
- if (items != null) {
- return !items.isEmpty();
+ String itemsExpression = pagingExpression(conn.getItemsJsonPath(),
conn.getItemsXPath());
+ String countingExpression =
+ Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+ if (Utils.isEmpty(countingExpression)) {
+ return !Utils.isEmpty(NVL(ex.body, "").trim());
+ }
+ List<String> items = splitResultItems(ex.body, countingExpression);
+ if (items.isEmpty()) {
+ return false;
+ }
+ state.itemsRead += items.size();
+
+ Long totalCount = readTotalCount(conn, ex.body);
+ if (totalCount != null && state.itemsRead >= totalCount) {
+ if (isDetailed()) {
+ logDetailed(
+ BaseMessages.getString(
+ PKG, "Rest.Log.PaginationTotalCountReached", state.itemsRead,
totalCount));
+ }
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Picks the JsonPath or the XPath of a pair of paging expressions, the way
the cursor and
+ * next-URL expressions are picked: the JsonPath when the response is read
as JSON, otherwise the
+ * XPath for an XML response. A JsonPath with no XPath next to it is always
used, so a mistake in
+ * it fails loudly rather than being ignored. Returns {@code null} when
neither applies.
+ */
+ private String pagingExpression(String jsonPath, String xPath) {
Review Comment:
**XPaths are silently dropped for non-XML application types.** When only an
XPath is set and the application type is e.g. `TEXT PLAIN` or `FORM URLENCODED`
(which people do use for XML APIs), this returns `null` without a log line.
Paging then falls back to the split path or the empty-body rule, which never
ends for an envelope. That goes against the javadoc's "fails loudly". Please
log a warning (or throw) when an XPath is set but cannot be used.
##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -927,6 +930,7 @@ protected void runPaginationLoop(Object[] rowData) throws
HopException {
connection != null ? connection.getPaginationType() :
RestPaginationType.NONE;
PaginationState state = new PaginationState();
state.effectiveLimit = resolvePagingLimit(connection);
+ state.pageNumber = resolveFirstPageNumber(connection);
Review Comment:
**The first page number is resolved for every paging type.** The editor
keeps the fields of the other paging types. So a connection that was once
`PAGE_NUMBER` with first page `${START_PAGE}` (or `-1`), and was later switched
to `OFFSET_LIMIT`, `CURSOR` or `LINK_HEADER`, fails with
`Rest.Error.PaginationFirstPageNumber` when `START_PAGE` isn't set, although
the field is ignored for that type. Please only call `resolveFirstPageNumber`
when the type is `PAGE_NUMBER`.
##########
plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnectionEditor.java:
##########
@@ -948,10 +985,60 @@ private void buildPaginationWidgets(Composite p) {
fdNx.left = new FormAttachment(middle, margin);
fdNx.right = new FormAttachment(100, -margin);
wNextPageUrlXPath.setLayoutData(fdNx);
+ lastControl = wNextPageUrlXPath;
+
+ wItemsJsonPath =
+ addPaginationTextRow(
+ p,
+ lastControl,
+ "RestConnectionEditor.Pagination.ItemsJson.Label",
+ "RestConnectionEditor.Pagination.ItemsJson.Tooltip");
+ wItemsXPath =
+ addPaginationTextRow(
+ p,
+ wItemsJsonPath,
+ "RestConnectionEditor.Pagination.ItemsXPath.Label",
Review Comment:
**The XPath rows reuse the JsonPath tooltips.** This XPath row (and the
Total count XPath row) uses the `...Json.Tooltip` key, which suggests `$.list`
and `$.count`. Please add `ItemsXPath`/`TotalCountXPath` tooltip keys with
XPath examples.
##########
plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnectionEditor.java:
##########
@@ -948,10 +985,60 @@ private void buildPaginationWidgets(Composite p) {
fdNx.left = new FormAttachment(middle, margin);
fdNx.right = new FormAttachment(100, -margin);
wNextPageUrlXPath.setLayoutData(fdNx);
+ lastControl = wNextPageUrlXPath;
+
+ wItemsJsonPath =
+ addPaginationTextRow(
+ p,
+ lastControl,
+ "RestConnectionEditor.Pagination.ItemsJson.Label",
+ "RestConnectionEditor.Pagination.ItemsJson.Tooltip");
+ wItemsXPath =
+ addPaginationTextRow(
+ p,
+ wItemsJsonPath,
+ "RestConnectionEditor.Pagination.ItemsXPath.Label",
+ "RestConnectionEditor.Pagination.ItemsJson.Tooltip");
+ wTotalCountJsonPath =
+ addPaginationTextRow(
+ p,
+ wItemsXPath,
+ "RestConnectionEditor.Pagination.TotalCountJson.Label",
+ "RestConnectionEditor.Pagination.TotalCountJson.Tooltip");
+ wTotalCountXPath =
+ addPaginationTextRow(
+ p,
+ wTotalCountJsonPath,
+ "RestConnectionEditor.Pagination.TotalCountXPath.Label",
+ "RestConnectionEditor.Pagination.TotalCountJson.Tooltip");
refreshPaginationSensitiveFields();
}
+ /** One label and text field below {@code above} on the Pagination tab. */
+ private TextVar addPaginationTextRow(
Review Comment:
**Six more hand-laid FormAttachment rows.** `addPaginationTextRow` adds six
of them, plus a hand-built ScrolledComposite with a min-height listener and
manual copying in `setWidgetsContent`/`getWidgetsContent`. The new
`RestConnection` fields have no `@GuiWidgetElement`. Our dialog convention is
annotated fields in a grouped `GuiCompositeWidgets` layout (tabs/boxes), which
also scrolls and resizes on its own. The editor was hand-built before this PR,
so I'm fine with doing this in a follow-up, but I'd rather not grow the
hand-built part further. At least the new pagination fields could be annotated.
##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
}
/**
- * @return whether this response appears to carry at least one item (whole
body when no
- * split-path, or extracted elements when splitting).
+ * Whether page, offset and cursor paging should ask for another page after
this one.
+ *
+ * <p>A page is empty when the connection's items path finds nothing in it.
Without an items path
+ * the result split path decides, and without either an empty body does —
which an API that wraps
+ * its results in an envelope never sends (#9549). The items path wins over
the split path because
+ * it describes the API, while the split path describes the rows this
transform wants.
+ *
+ * <p>With a total count path as well, paging also stops once that many
items were read, so the
+ * last, empty, page is not fetched at all.
*/
- private boolean exchangeHadPayload(RestExchangeResult ex, String
resolvedSplitPath)
+ private boolean pageHadItemsAndMoreRemain(
+ RestConnection conn, PaginationState state, RestExchangeResult ex,
String resolvedSplitPath)
throws HopException {
- List<String> items =
- Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body,
resolvedSplitPath);
- if (items != null) {
- return !items.isEmpty();
+ String itemsExpression = pagingExpression(conn.getItemsJsonPath(),
conn.getItemsXPath());
+ String countingExpression =
+ Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+ if (Utils.isEmpty(countingExpression)) {
+ return !Utils.isEmpty(NVL(ex.body, "").trim());
+ }
+ List<String> items = splitResultItems(ex.body, countingExpression);
Review Comment:
**An items XPath that points at the container never sees an empty page.**
With the XPath counterpart of the documented `$.list` example, e.g.
`/envelope/list`, the page past the end is
`<envelope><count>6</count><list/></envelope>`. `splitResultItems` returns one
node, which serialises to `<list/>`, not blank, so the page counts as 1 item
and paging runs to `maxPagesLoops`. So the endless paging of #9549 is still
there for XML. `itemsRead` also grows by 1 per page instead of by the item
count, which breaks the total-count stop. JSON `$.list` works only because
JsonPath returns an `Iterable`. For XPath, count the container's element
children (or document that the path must select the items, e.g.
`/envelope/list/item`).
##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
}
/**
- * @return whether this response appears to carry at least one item (whole
body when no
- * split-path, or extracted elements when splitting).
+ * Whether page, offset and cursor paging should ask for another page after
this one.
+ *
+ * <p>A page is empty when the connection's items path finds nothing in it.
Without an items path
+ * the result split path decides, and without either an empty body does —
which an API that wraps
+ * its results in an envelope never sends (#9549). The items path wins over
the split path because
+ * it describes the API, while the split path describes the rows this
transform wants.
+ *
+ * <p>With a total count path as well, paging also stops once that many
items were read, so the
+ * last, empty, page is not fetched at all.
*/
- private boolean exchangeHadPayload(RestExchangeResult ex, String
resolvedSplitPath)
+ private boolean pageHadItemsAndMoreRemain(
+ RestConnection conn, PaginationState state, RestExchangeResult ex,
String resolvedSplitPath)
throws HopException {
- List<String> items =
- Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body,
resolvedSplitPath);
- if (items != null) {
- return !items.isEmpty();
+ String itemsExpression = pagingExpression(conn.getItemsJsonPath(),
conn.getItemsXPath());
+ String countingExpression =
+ Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+ if (Utils.isEmpty(countingExpression)) {
+ return !Utils.isEmpty(NVL(ex.body, "").trim());
+ }
+ List<String> items = splitResultItems(ex.body, countingExpression);
+ if (items.isEmpty()) {
+ return false;
+ }
+ state.itemsRead += items.size();
+
+ Long totalCount = readTotalCount(conn, ex.body);
+ if (totalCount != null && state.itemsRead >= totalCount) {
+ if (isDetailed()) {
+ logDetailed(
+ BaseMessages.getString(
+ PKG, "Rest.Log.PaginationTotalCountReached", state.itemsRead,
totalCount));
+ }
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Picks the JsonPath or the XPath of a pair of paging expressions, the way
the cursor and
+ * next-URL expressions are picked: the JsonPath when the response is read
as JSON, otherwise the
+ * XPath for an XML response. A JsonPath with no XPath next to it is always
used, so a mistake in
+ * it fails loudly rather than being ignored. Returns {@code null} when
neither applies.
+ */
+ private String pagingExpression(String jsonPath, String xPath) {
+ String appType = NVL(resolve(meta.getApplicationType()), "");
+ String json = resolveSplitPathOrPagingExpression(jsonPath);
+ String xml = resolveSplitPathOrPagingExpression(xPath);
+ if (!Utils.isEmpty(json) && (shouldParseResponseAsJson(appType, json) ||
Utils.isEmpty(xml))) {
+ return json;
+ }
+ if (!Utils.isEmpty(xml) && isXmlApplicationType(appType)) {
+ return xml;
+ }
+ return null;
+ }
+
+ private static boolean isXmlApplicationType(String appType) {
Review Comment:
**The XML type list now exists twice.** `isXmlApplicationType()` copies the
five-type check that `splitResultItems()` still spells out inline. If a type is
added to only one of them, `pagingExpression()` picks the XPath while
`splitResultItems()` throws, or the other way round. Please have
`splitResultItems()` call this method.
##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
}
/**
- * @return whether this response appears to carry at least one item (whole
body when no
- * split-path, or extracted elements when splitting).
+ * Whether page, offset and cursor paging should ask for another page after
this one.
+ *
+ * <p>A page is empty when the connection's items path finds nothing in it.
Without an items path
+ * the result split path decides, and without either an empty body does —
which an API that wraps
+ * its results in an envelope never sends (#9549). The items path wins over
the split path because
+ * it describes the API, while the split path describes the rows this
transform wants.
+ *
+ * <p>With a total count path as well, paging also stops once that many
items were read, so the
+ * last, empty, page is not fetched at all.
*/
- private boolean exchangeHadPayload(RestExchangeResult ex, String
resolvedSplitPath)
+ private boolean pageHadItemsAndMoreRemain(
+ RestConnection conn, PaginationState state, RestExchangeResult ex,
String resolvedSplitPath)
throws HopException {
- List<String> items =
- Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body,
resolvedSplitPath);
- if (items != null) {
- return !items.isEmpty();
+ String itemsExpression = pagingExpression(conn.getItemsJsonPath(),
conn.getItemsXPath());
+ String countingExpression =
+ Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+ if (Utils.isEmpty(countingExpression)) {
+ return !Utils.isEmpty(NVL(ex.body, "").trim());
+ }
+ List<String> items = splitResultItems(ex.body, countingExpression);
+ if (items.isEmpty()) {
+ return false;
+ }
+ state.itemsRead += items.size();
+
+ Long totalCount = readTotalCount(conn, ex.body);
+ if (totalCount != null && state.itemsRead >= totalCount) {
+ if (isDetailed()) {
+ logDetailed(
+ BaseMessages.getString(
+ PKG, "Rest.Log.PaginationTotalCountReached", state.itemsRead,
totalCount));
+ }
+ return false;
+ }
+ return true;
+ }
+
+ /**
+ * Picks the JsonPath or the XPath of a pair of paging expressions, the way
the cursor and
+ * next-URL expressions are picked: the JsonPath when the response is read
as JSON, otherwise the
+ * XPath for an XML response. A JsonPath with no XPath next to it is always
used, so a mistake in
+ * it fails loudly rather than being ignored. Returns {@code null} when
neither applies.
+ */
+ private String pagingExpression(String jsonPath, String xPath) {
+ String appType = NVL(resolve(meta.getApplicationType()), "");
+ String json = resolveSplitPathOrPagingExpression(jsonPath);
+ String xml = resolveSplitPathOrPagingExpression(xPath);
+ if (!Utils.isEmpty(json) && (shouldParseResponseAsJson(appType, json) ||
Utils.isEmpty(xml))) {
+ return json;
+ }
+ if (!Utils.isEmpty(xml) && isXmlApplicationType(appType)) {
+ return xml;
+ }
+ return null;
+ }
+
+ private static boolean isXmlApplicationType(String appType) {
+ return RestMeta.APPLICATION_TYPE_XML.equals(appType)
+ || RestMeta.APPLICATION_TYPE_TEXT_XML.equals(appType)
+ || RestMeta.APPLICATION_TYPE_ATOM_XML.equals(appType)
+ || RestMeta.APPLICATION_TYPE_SVG_XML.equals(appType)
+ || RestMeta.APPLICATION_TYPE_XHTML.equals(appType);
+ }
+
+ /**
+ * The total number of items the API reports through the connection's total
count path, or {@code
+ * null} when no path is set or the page does not hold a whole number there.
+ */
+ private Long readTotalCount(RestConnection conn, String body) throws
HopException {
+ String expression = pagingExpression(conn.getTotalCountJsonPath(),
conn.getTotalCountXPath());
+ if (Utils.isEmpty(expression)) {
+ return null;
+ }
+ String value;
+ try {
+ // The same JSON-or-XML rule as splitResultItems(), so the count and the
items agree.
+ if (shouldParseResponseAsJson(NVL(resolve(meta.getApplicationType()),
""), expression)) {
+ Object raw =
+ JsonPath.using(JSON_PATH_CONFIGURATION).parse(NVL(body,
"{}")).read(expression);
+ if (raw instanceof Number number) {
+ return number.longValue();
+ }
+ value = raw == null ? null : raw.toString();
+ } else {
+ Document doc;
+ try (StringReader sr = new StringReader(NVL(body, ""))) {
+ doc = documentBuilderFactory().newDocumentBuilder().parse(new
InputSource(sr));
Review Comment:
**A body that can't be parsed fails the whole transform.** A `SAXException`
here, or a JSON body that JsonPath can't parse (`SUPPRESS_EXCEPTIONS` doesn't
cover `parse()`), becomes a fatal `Rest.Error.PaginationTotalCount`. A missing
or non-numeric count a few lines further down only logs and continues. I'd
treat a parse failure the same way: log it and continue as if there were no
count.
##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
}
/**
- * @return whether this response appears to carry at least one item (whole
body when no
- * split-path, or extracted elements when splitting).
+ * Whether page, offset and cursor paging should ask for another page after
this one.
+ *
+ * <p>A page is empty when the connection's items path finds nothing in it.
Without an items path
+ * the result split path decides, and without either an empty body does —
which an API that wraps
+ * its results in an envelope never sends (#9549). The items path wins over
the split path because
+ * it describes the API, while the split path describes the rows this
transform wants.
+ *
+ * <p>With a total count path as well, paging also stops once that many
items were read, so the
+ * last, empty, page is not fetched at all.
*/
- private boolean exchangeHadPayload(RestExchangeResult ex, String
resolvedSplitPath)
+ private boolean pageHadItemsAndMoreRemain(
+ RestConnection conn, PaginationState state, RestExchangeResult ex,
String resolvedSplitPath)
throws HopException {
- List<String> items =
- Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body,
resolvedSplitPath);
- if (items != null) {
- return !items.isEmpty();
+ String itemsExpression = pagingExpression(conn.getItemsJsonPath(),
conn.getItemsXPath());
Review Comment:
**The items path and the split path can disagree.** The items path decides
whether a page is empty, but `emitPagedResultRows` still emits rows by the
split path. With items `$.list` and split `$.data[*]` (a mistake or an API
change), paging continues while every page emits 0 rows, and at the end
`enforcePagingSplitEmittedSomething` throws `PagingSplitProducedZeroRows`,
pointing at the split path rather than the mismatch. In the reverse case,
paging stops after the first page while the split path would have found rows.
Could the stop rule use the split path's result when both are set, or at least
report the mismatch clearly?
##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
}
/**
- * @return whether this response appears to carry at least one item (whole
body when no
- * split-path, or extracted elements when splitting).
+ * Whether page, offset and cursor paging should ask for another page after
this one.
+ *
+ * <p>A page is empty when the connection's items path finds nothing in it.
Without an items path
+ * the result split path decides, and without either an empty body does —
which an API that wraps
+ * its results in an envelope never sends (#9549). The items path wins over
the split path because
+ * it describes the API, while the split path describes the rows this
transform wants.
+ *
+ * <p>With a total count path as well, paging also stops once that many
items were read, so the
+ * last, empty, page is not fetched at all.
*/
- private boolean exchangeHadPayload(RestExchangeResult ex, String
resolvedSplitPath)
+ private boolean pageHadItemsAndMoreRemain(
+ RestConnection conn, PaginationState state, RestExchangeResult ex,
String resolvedSplitPath)
throws HopException {
- List<String> items =
- Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body,
resolvedSplitPath);
- if (items != null) {
- return !items.isEmpty();
+ String itemsExpression = pagingExpression(conn.getItemsJsonPath(),
conn.getItemsXPath());
+ String countingExpression =
+ Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+ if (Utils.isEmpty(countingExpression)) {
Review Comment:
**The total count path is ignored here.** With a Total count path
(`$.count`) but no items path and no result split path, `countingExpression` is
empty and we return on the non-empty-body check before `readTotalCount()` runs.
An envelope API never sends an empty body, so paging runs to `maxPagesLoops`
and emits duplicate envelope rows, and the user gets no hint that the setting
had no effect. Either apply the total count here too, or log a warning (or fail
validation) that a total count needs an items path.
##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
}
/**
- * @return whether this response appears to carry at least one item (whole
body when no
- * split-path, or extracted elements when splitting).
+ * Whether page, offset and cursor paging should ask for another page after
this one.
+ *
+ * <p>A page is empty when the connection's items path finds nothing in it.
Without an items path
+ * the result split path decides, and without either an empty body does —
which an API that wraps
+ * its results in an envelope never sends (#9549). The items path wins over
the split path because
+ * it describes the API, while the split path describes the rows this
transform wants.
+ *
+ * <p>With a total count path as well, paging also stops once that many
items were read, so the
+ * last, empty, page is not fetched at all.
*/
- private boolean exchangeHadPayload(RestExchangeResult ex, String
resolvedSplitPath)
+ private boolean pageHadItemsAndMoreRemain(
Review Comment:
**Each page body is parsed several times.** It is parsed up to 4 times per
page: the split in `emitPagedResultRows`, the items path, the total count and
the cursor. On top of that, `pagingExpression()` builds
`substitutionMapSnapshot()` twice per call, twice per page. For pages with
thousands of items that's wasted CPU and memory in the paging loop. Suggest
resolving the items and count expressions once in `runPaginationLoop`, and
parsing the body once per page.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]