hansva commented on code in PR #9574:
URL: https://github.com/apache/hop/pull/9574#discussion_r4614856200
##########
plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnectionEditor.java:
##########
@@ -1512,6 +1575,18 @@ public void setWidgetsContent() {
wProxyPassword.setText(Const.NVL(metadata.getProxyPassword(), ""));
wNonProxyHosts.setText(Const.NVL(metadata.getNonProxyHosts(), ""));
+ wHeaders.removeAll();
+ if (metadata.getHeaders() != null) {
+ for (RestConnectionHeader header : metadata.getHeaders()) {
+ TableItem item = new TableItem(wHeaders.table, SWT.NONE);
+ item.setText(1, Const.NVL(header.getName(), ""));
Review Comment:
**NPE on a null header entry.** `resolveStaticHeaders` skips null entries,
but this loop dereferences `header` directly. With `"headers":[null]` in a
hand-edited or partially loaded JSON, the connection runs fine but the editor
throws, so the connection can't be edited. Please skip nulls here too.
##########
plugins/transforms/httppost/src/main/java/org/apache/hop/pipeline/transforms/httppost/HttpPost.java:
##########
@@ -142,6 +142,10 @@ Object[] callHttpPOST(Object[] rowData) throws
HopException {
}
// add HttpPost headers
addHttpHeaders(post, rowData);
+ // The connection's static headers come after the transform's own: a
header set here wins.
+ if (data.restAuthenticator != null) {
+ data.restAuthenticator.applyStaticHeaders(post, uri.toString());
Review Comment:
**The same request-decoration code is copied into Http, HttpPost and
ActionHttp.** Each copy does: apply the auth headers into a temp map,
`setHeader` each entry, then `applyStaticHeaders`. In each copy the static
headers are applied at a different point; here it's a separate block with its
own null check, in the other two it's inside `addConnectionAuthentication`. A
single `RestAuthenticator.applyTo(HttpMessage request, String uri)` would
remove the copies, and would also be a natural place for the 401 renewal (see
my comment on `Http.java`).
##########
plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/client/RestAuthenticator.java:
##########
@@ -72,6 +113,100 @@ public void applyRequestHeaders(Map<String, String>
headers, String requestUri)
}
}
+ /**
+ * Adds the connection's static headers (issue #9551) that the map does not
already carry.
+ *
+ * @param headers the outbound header map, modified in place
+ * @param requestUri the URI this request is going to
+ */
+ public void applyStaticHeaders(Map<String, String> headers, String
requestUri) {
+ if (headers == null || !hasStaticHeaders() ||
!targetMatchesOrigin(requestUri)) {
+ return;
+ }
+ for (Map.Entry<String, String> header :
settings.getStaticHeaders().entrySet()) {
+ if (!headerAlreadySupplied(headers, header.getKey())) {
Review Comment:
**A static `Accept` header on the connection is always dropped.** The REST
transform puts its default `Accept` (from `data.mediaType`) into `headerMap`
(`Rest.java` ~line 392), before the static headers are applied, so
`headerAlreadySupplied` is true and the connection's `Accept:
application/vnd.api+json` is skipped. HTTP POST does the same with its default
`Content-Type: text/xml` via `addHeadersContentType`. The editor says only a
header "set on the transform or by an input row" wins, so the user gets no
indication. Please let a static header beat the transform's *default*: only a
header the user set explicitly should win.
##########
plugins/transforms/http/src/main/java/org/apache/hop/pipeline/transforms/http/Http.java:
##########
@@ -312,17 +312,19 @@ private CloseableHttpClient httpClient(HttpHost target) {
/**
* Bearer, API-key and preemptive Basic authentication are request headers
rather than answers to
* a challenge, so the connection writes them onto every request itself.
They stay scoped to the
- * connection's base URL: an absolute URL on another host is sent without
them.
+ * connection's base URL: an absolute URL on another host is sent without
them. The connection's
+ * static headers follow the same rule, but a header this transform already
set wins.
*/
private void addConnectionAuthentication(HttpGet method, URI uri) {
if (data.restAuthenticator == null) {
return;
}
Map<String, String> authHeaders = new LinkedHashMap<>();
- data.restAuthenticator.applyRequestHeaders(authHeaders, uri.toString());
+ data.restAuthenticator.applyAuthenticationHeaders(authHeaders,
uri.toString());
for (Map.Entry<String, String> authHeader : authHeaders.entrySet()) {
method.setHeader(authHeader.getKey(), authHeader.getValue());
}
+ data.restAuthenticator.applyStaticHeaders(method, uri.toString());
Review Comment:
**The 401 token renewal only covers the REST transform and the connection
test.** This transform, HTTP POST and the HTTP action now use the same OAuth 2
REST connection, but none of them calls `refreshAfterUnauthorized` /
`invalidate`. When the provider revokes the token early (another client fetched
a new one), every row here gets a 401 until the cached expiry passes, which is
exactly the #9550 bug. I'd prefer one fix in the shared client layer, e.g. an
exec-chain interceptor in `RestClientFactory` or a shared `RestAuthenticator`
execute helper, so all four callers get it.
##########
plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnectionEditor.java:
##########
@@ -666,6 +686,49 @@ public void widgetSelected(SelectionEvent e) {
enableControls(controls);
}
+ /** Builds the table of headers sent on every request (issue #9551) on the
Headers tab. */
+ private void buildHeadersWidgets(Composite p) {
+ Label wlInfo = new Label(p, SWT.LEFT | SWT.WRAP);
+ wlInfo.setText(BaseMessages.getString(PKG,
"RestConnectionEditor.Headers.Info"));
+ PropsUi.setLook(wlInfo);
+ FormData fdlInfo = new FormData();
+ fdlInfo.left = new FormAttachment(0, 0);
+ fdlInfo.right = new FormAttachment(100, 0);
+ fdlInfo.top = new FormAttachment(0, margin);
+ wlInfo.setLayoutData(fdlInfo);
+
+ ColumnInfo nameColumn =
+ new ColumnInfo(
+ BaseMessages.getString(PKG,
"RestConnectionEditor.Headers.Column.Name"),
+ ColumnInfo.COLUMN_TYPE_TEXT,
+ false,
+ false);
+ nameColumn.setUsingVariables(true);
+ ColumnInfo valueColumn =
Review Comment:
**Header values are plain text.** The typical use for static headers is
something credential-like (`x-api-key`, `Authorization: Basic …`). The values
are shown unmasked and saved as typed into
`metadata/rest-connection/<name>.json`, while the connection's other secrets
use `PasswordTextVar` and are encrypted on save. `resolveStaticHeaders` already
understands `Encrypted …` values. Could the editor encrypt values on save (or
at least offer a "secret" flag per header and mask those)?
##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -523,6 +532,82 @@ protected RestExchangeResult invokeRestExchange(
// The client is not closed here: it is shared by every row and released
on dispose().
}
+ /**
+ * Sends the request, and when it comes back 401 on an OAuth 2 token Hop
applied, sends it once
+ * more with a new token (issue #9550). Without this a token the server
stopped accepting before
+ * its expiry time — revoked, rotated, or replaced because another client
asked for one — keeps
+ * being sent until that time passes, and adding 401 to the retry status
codes only resends it.
+ *
+ * <p>It runs inside {@link #executeWithRetry}, so the new token is in place
before any configured
+ * retry. A streaming request reports a failed status as an exception rather
than a response; a
+ * 401 there is just as safe to resend, because no record has gone
downstream yet.
+ */
+ private HttpExchange sendRenewingRejectedToken(
+ Supplier<HttpExchange> send,
+ Map<String, String> headers,
+ String requestUri,
+ boolean[] tokenRefreshed) {
+ HttpExchange exchange;
+ try {
+ exchange = send.get();
+ } catch (HopRuntimeException e) {
+ if (isUnauthorizedStream(e) && renewRejectedToken(headers, requestUri,
tokenRefreshed)) {
+ return send.get();
+ }
+ throw e;
+ }
+ if (exchange.status() == HttpStatus.SC_UNAUTHORIZED
+ && renewRejectedToken(headers, requestUri, tokenRefreshed)) {
+ return send.get();
Review Comment:
**A resent request is counted twice.** `send.get()` goes through
`executeRequest` again, which calls `trackRequestBytes` again. A 10 MB POST
resent once after a 401 reports 20 MB of request bytes for one row. The
counters should stay exact: count the resend separately, or don't count it.
##########
plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnection.java:
##########
@@ -407,30 +439,61 @@ private boolean authTypeEquals(String canonicalLabel) {
&& authType.trim().equalsIgnoreCase(canonicalLabel.trim());
}
- /** Performs a GET against the URL with this connection's authentication,
returning the body. */
+ /**
+ * Performs a GET against the URL with this connection's authentication and
static headers,
+ * returning the body. An OAuth 2 token the server rejects with a 401 is
replaced and the request
+ * sent once more (issue #9550).
+ */
public String getResponse(String url) throws HopException {
RestClientSettings settings = createClientSettings();
+ // Scoped to the URL rather than to the base URL: see
applyAuthentication().
+ RestClientSettings requestSettings = createClientSettings();
+ requestSettings.setAuthOrigin(url);
+ RestAuthenticator authenticator = new RestAuthenticator(requestSettings);
Map<String, String> headers = new LinkedHashMap<>();
- applyAuthentication(headers, url);
-
- HttpUriRequestBase request = new HttpUriRequestBase("GET",
URI.create(url));
- headers.forEach(request::addHeader);
+ authenticator.applyRequestHeaders(headers, url);
try (CloseableHttpClient httpClient =
RestClientFactory.createClient(settings)) {
- return httpClient.execute(
- request,
- response -> {
- if (response.getCode() != HttpStatus.SC_OK) {
- throw new IOException("Error connecting to " + url + ": " +
response.getCode());
- }
- HttpEntity entity = response.getEntity();
- return entity == null ? "" : EntityUtils.toString(entity);
- });
+ Response response = get(httpClient, url, headers);
+ if (response.status() == HttpStatus.SC_UNAUTHORIZED
+ && authenticator.refreshAfterUnauthorized(headers, url)) {
+ getLog().logBasic("The server rejected the OAuth 2 token; retrying
with a new token.");
Review Comment:
**Hard-coded English log messages.** This log line, and the `Error
connecting to …` text below it, aren't translatable. `Rest.java` logs the same
event through the `Rest.Log.OAuth2TokenRejectedRetrying` key. Please use
`BaseMessages` here too.
##########
plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnectionEditor.java:
##########
@@ -666,6 +686,49 @@ public void widgetSelected(SelectionEvent e) {
enableControls(controls);
}
+ /** Builds the table of headers sent on every request (issue #9551) on the
Headers tab. */
+ private void buildHeadersWidgets(Composite p) {
+ Label wlInfo = new Label(p, SWT.LEFT | SWT.WRAP);
+ wlInfo.setText(BaseMessages.getString(PKG,
"RestConnectionEditor.Headers.Info"));
+ PropsUi.setLook(wlInfo);
+ FormData fdlInfo = new FormData();
+ fdlInfo.left = new FormAttachment(0, 0);
+ fdlInfo.right = new FormAttachment(100, 0);
+ fdlInfo.top = new FormAttachment(0, margin);
+ wlInfo.setLayoutData(fdlInfo);
+
+ ColumnInfo nameColumn =
+ new ColumnInfo(
+ BaseMessages.getString(PKG,
"RestConnectionEditor.Headers.Column.Name"),
+ ColumnInfo.COLUMN_TYPE_TEXT,
+ false,
+ false);
+ nameColumn.setUsingVariables(true);
+ ColumnInfo valueColumn =
+ new ColumnInfo(
+ BaseMessages.getString(PKG,
"RestConnectionEditor.Headers.Column.Value"),
+ ColumnInfo.COLUMN_TYPE_TEXT,
+ false,
+ false);
+ valueColumn.setUsingVariables(true);
+
+ wHeaders =
+ new TableView(
Review Comment:
**The Headers tab is hand-laid out.** It's built with FormAttachment and a
free-standing `TableView`. Our dialog convention is annotated fields in a
grouped `GuiCompositeWidgets` layout, with a `TableView` added through
`GuiCompositeWidgets.registerExtraGroup(...)` rather than laid out by hand. The
editor was already hand-built, so this can be a follow-up, but I'd rather not
add new hand-laid tabs.
##########
plugins/misc/rest/src/main/java/org/apache/hop/metadata/rest/RestConnection.java:
##########
@@ -407,30 +439,61 @@ private boolean authTypeEquals(String canonicalLabel) {
&& authType.trim().equalsIgnoreCase(canonicalLabel.trim());
}
- /** Performs a GET against the URL with this connection's authentication,
returning the body. */
+ /**
+ * Performs a GET against the URL with this connection's authentication and
static headers,
+ * returning the body. An OAuth 2 token the server rejects with a 401 is
replaced and the request
+ * sent once more (issue #9550).
+ */
public String getResponse(String url) throws HopException {
RestClientSettings settings = createClientSettings();
+ // Scoped to the URL rather than to the base URL: see
applyAuthentication().
+ RestClientSettings requestSettings = createClientSettings();
Review Comment:
**`createClientSettings()` runs twice per `getResponse`.** Each call
resolves the variables again, resolves and decrypts the static headers, and
rebuilds the SSL context, which loads the keystore and truststore from disk. So
a connection test with client certificates loads both stores twice. Derive
`requestSettings` from `settings` (copy + `setAuthOrigin(url)`) instead.
Related: the public `applyAuthentication` now has no callers. `getResponse`
re-implements this same sequence inline (`createClientSettings` +
`setAuthOrigin(url)` + `new RestAuthenticator` + apply headers), so there are
two copies of the URL-scoped auth logic. A future change to origin scoping has
to be made in both. Please extract a private helper both use, or drop this
method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]