This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new b69e05004b Issue #6982 : Add SAS token authentication to the Azure VFS
plugin (#7936)
b69e05004b is described below
commit b69e05004bcfe6488c96549701afda2e589c015b
Author: vbhanuchander-lang <[email protected]>
AuthorDate: Thu Aug 13 09:42:15 2026 -0400
Issue #6982 : Add SAS token authentication to the Azure VFS plugin (#7936)
The Azure connection could authenticate with a storage account key or a
managed identity. An account key grants full, non-expiring access to the
whole storage account, which rules out Hop for deployments that have to
enforce least privilege.
Adds "SAS Token" as a third authentication type. A shared access signature
is scoped to specific resources and permissions and carries an expiry, and
is what Microsoft recommends for application level access to Storage.
- AzureMetadataType: new sasToken property, marked password = true so it is
encrypted at rest and masked in the UI, matching the account key.
- AzureFileProvider: a "SAS Token" branch passing the token to
DataLakeServiceClientBuilder.sasToken. The SDK accepts the query string
with or without a leading '?', so nothing is normalised here.
- AzureMetadataTypeEditor: the new option, plus a password field that is
shown only for that authentication type, mirroring the account key field.
- Documented the three authentication types, which were previously not
described on the Azure VFS page at all.
Existing connections are untouched: the constructor still defaults to
"Key" and the provider still treats an empty authentication type as "Key".
---
.../ROOT/pages/vfs/azure-blob-storage-vfs.adoc | 39 ++++++++++
.../apache/hop/vfs/azure/AzureFileProvider.java | 14 ++++
.../vfs/azure/metadatatype/AzureMetadataType.java | 21 ++++++
.../metadatatype/AzureMetadataTypeEditor.java | 36 +++++++++-
.../messages/messages_en_US.properties | 2 +
.../vfs/azure/AzureSasTokenAuthenticationTest.java | 84 ++++++++++++++++++++++
6 files changed, 195 insertions(+), 1 deletion(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
b/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
index 3fcb3fc7ff..8b5519837e 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/vfs/azure-blob-storage-vfs.adoc
@@ -68,6 +68,45 @@ Once done you will see an `azure` entry in the central
`hop-config.json` file:
}
----
+== Authentication types
+
+An Azure connection in the metadata perspective can authenticate in three
ways, selected with the
+*Authentication Type* option. Connections that predate this option keep
working: an empty
+authentication type is treated as `Key`.
+
+[options="header"]
+|===
+|Authentication Type|What you provide|When to use it
+|`Key`|The storage account key|Simplest to set up. The key grants full,
non-expiring access to the whole storage account.
+|`Managed Identity`|Nothing|Runs on Azure with a managed identity, or locally
through the Azure CLI (`az login`) or the `AZURE_CLIENT_ID` / `AZURE_TENANT_ID`
/ `AZURE_CLIENT_SECRET` environment variables. The identity needs a role such
as `Storage Blob Data Contributor`.
+|`SAS Token`|A shared access signature|Access is limited to the resources,
permissions and expiry encoded in the signature, so it avoids handing out a
long lived account key.
+|===
+
+=== SAS token
+
+A shared access signature is the approach Microsoft recommends for application
level access to
+Azure Storage, because it can be scoped and given an expiry rather than
granting unrestricted
+access to the account.
+
+Generate one in the Azure portal under *Shared access signature* for the
storage account, or with
+`az storage account generate-sas`, then paste the *query string* into the *SAS
Token* field:
+
+[source]
+----
+sv=2022-11-02&ss=b&srt=co&sp=rl&se=2026-12-31T00:00:00Z&sig=...
+----
+
+A leading `?` is accepted and may be left in place. Do not paste a full URL —
only the query
+string portion is expected.
+
+The token is stored as a password in the connection metadata, which means it
is encrypted at rest
+and masked in the UI, and it can be supplied through a variable so it does not
have to be committed
+to a project at all.
+
+NOTE: A signature stops working once it expires, and Hop reports the resulting
failure as an
+authentication error. Renew the token, or use `Managed Identity` where you
want credentials that
+rotate for you.
+
== Usage and testing
To test if the configuration works you can simply upload a small CSV file in
an Azure Storage folder and then use File/Open in Hop GUI.
diff --git
a/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure/AzureFileProvider.java
b/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure/AzureFileProvider.java
index 1c3ae3d369..9f0733e12e 100644
---
a/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure/AzureFileProvider.java
+++
b/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure/AzureFileProvider.java
@@ -157,6 +157,20 @@ public class AzureFileProvider extends
AbstractOriginatingFileProvider {
+ "Also ensure your identity has proper permissions (e.g.,
'Storage Blob Data Contributor' role) on the storage account.",
e);
}
+ } else if ("SAS Token".equals(authType)) {
+ // Use a shared access signature, which is scoped and time limited
+ if (StringUtils.isEmpty(azureMetadataType.getSasToken())) {
+ throw new FileSystemException(
+ "Azure configuration \""
+ + azureMetadataType.getName()
+ + "\" is missing a SAS token");
+ }
+
+ String sasToken =
+ Encr.decryptPasswordOptionallyEncrypted(
+ variables.resolve(azureMetadataType.getSasToken()));
+
+ serviceClient = clientBuilder.sasToken(sasToken).buildClient();
} else {
// Use Key-based authentication
if (StringUtils.isEmpty(azureMetadataType.getStorageAccountKey())) {
diff --git
a/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure/metadatatype/AzureMetadataType.java
b/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure/metadatatype/AzureMetadataType.java
index 535c8fbefb..e6c82ee407 100644
---
a/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure/metadatatype/AzureMetadataType.java
+++
b/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure/metadatatype/AzureMetadataType.java
@@ -46,6 +46,9 @@ public class AzureMetadataType extends HopMetadataBase
implements Serializable,
@HopMetadataProperty(password = true)
private String storageAccountKey;
+ @HopMetadataProperty(password = true)
+ private String sasToken;
+
@HopMetadataProperty private String storageAccountEndpoint;
/** Cache TTL in seconds for list-result caching (same as S3/MinIO). */
@@ -55,4 +58,22 @@ public class AzureMetadataType extends HopMetadataBase
implements Serializable,
this.authenticationType = "Key"; // Default to Key authentication
this.cacheTtlSeconds = "5";
}
+
+ /**
+ * Gets the shared access signature token, used when the authentication type
is "SAS Token".
+ *
+ * @return the SAS token
+ */
+ public String getSasToken() {
+ return sasToken;
+ }
+
+ /**
+ * Sets the shared access signature token.
+ *
+ * @param sasToken the SAS token to set
+ */
+ public void setSasToken(String sasToken) {
+ this.sasToken = sasToken;
+ }
}
diff --git
a/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure/metadatatype/AzureMetadataTypeEditor.java
b/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure/metadatatype/AzureMetadataTypeEditor.java
index f7e5867ce1..f0019bbbc0 100644
---
a/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure/metadatatype/AzureMetadataTypeEditor.java
+++
b/plugins/tech/azure/src/main/java/org/apache/hop/vfs/azure/metadatatype/AzureMetadataTypeEditor.java
@@ -47,6 +47,8 @@ public class AzureMetadataTypeEditor extends
MetadataEditor<AzureMetadataType> {
private ComboVar wAuthenticationType;
private Label wlStorageAccountKey;
private PasswordTextVar wStorageAccountKey;
+ private Label wlSasToken;
+ private PasswordTextVar wSasToken;
private TextVar wStorageAccountEndpoint;
private TextVar wCacheTtlSeconds;
@@ -156,7 +158,7 @@ public class AzureMetadataTypeEditor extends
MetadataEditor<AzureMetadataType> {
wlAuthenticationType.setLayoutData(fdlAuthenticationType);
wAuthenticationType = new ComboVar(getVariables(), parent, SWT.SINGLE |
SWT.LEFT | SWT.BORDER);
PropsUi.setLook(wAuthenticationType);
- wAuthenticationType.setItems(new String[] {"Key", "Managed Identity"});
+ wAuthenticationType.setItems(new String[] {"Key", "Managed Identity", "SAS
Token"});
FormData fdAuthenticationType = new FormData();
fdAuthenticationType.top = new FormAttachment(wlAuthenticationType, 0,
SWT.CENTER);
fdAuthenticationType.left = new FormAttachment(middle, 0);
@@ -185,6 +187,29 @@ public class AzureMetadataTypeEditor extends
MetadataEditor<AzureMetadataType> {
wStorageAccountKey.setLayoutData(fdStorageAccountKey);
lastControl = wStorageAccountKey;
+ // The SAS token
+ //
+ wlSasToken = new Label(parent, SWT.RIGHT);
+ PropsUi.setLook(wlSasToken);
+ wlSasToken.setText(BaseMessages.getString(PKG,
"AzureMetadataTypeEditor.SasToken.Label"));
+ wlSasToken.setToolTipText(
+ BaseMessages.getString(PKG,
"AzureMetadataTypeEditor.SasToken.ToolTip"));
+ FormData fdlSasToken = new FormData();
+ fdlSasToken.top = new FormAttachment(lastControl, margin);
+ fdlSasToken.left = new FormAttachment(0, 0);
+ fdlSasToken.right = new FormAttachment(middle, -margin);
+ wlSasToken.setLayoutData(fdlSasToken);
+ wSasToken = new PasswordTextVar(getVariables(), parent, SWT.SINGLE |
SWT.LEFT | SWT.BORDER);
+ PropsUi.setLook(wSasToken);
+ wSasToken.setToolTipText(
+ BaseMessages.getString(PKG,
"AzureMetadataTypeEditor.SasToken.ToolTip"));
+ FormData fdSasToken = new FormData();
+ fdSasToken.top = new FormAttachment(wlSasToken, 0, SWT.CENTER);
+ fdSasToken.left = new FormAttachment(middle, 0);
+ fdSasToken.right = new FormAttachment(95, 0);
+ wSasToken.setLayoutData(fdSasToken);
+ lastControl = wSasToken;
+
// Cache TTL (seconds)
//
Label wlCacheTtlSeconds = new Label(parent, SWT.RIGHT);
@@ -210,8 +235,11 @@ public class AzureMetadataTypeEditor extends
MetadataEditor<AzureMetadataType> {
e -> {
String authType = wAuthenticationType.getText();
boolean showKey = "Key".equals(authType);
+ boolean showSasToken = "SAS Token".equals(authType);
wlStorageAccountKey.setVisible(showKey);
wStorageAccountKey.setVisible(showKey);
+ wlSasToken.setVisible(showSasToken);
+ wSasToken.setVisible(showSasToken);
parent.layout(true, true);
});
@@ -223,6 +251,7 @@ public class AzureMetadataTypeEditor extends
MetadataEditor<AzureMetadataType> {
wStorageAccountName.addModifyListener(e -> setChanged());
wAuthenticationType.addModifyListener(e -> setChanged());
wStorageAccountKey.addModifyListener(e -> setChanged());
+ wSasToken.addModifyListener(e -> setChanged());
wStorageAccountEndpoint.addModifyListener(e -> setChanged());
wCacheTtlSeconds.addModifyListener(e -> setChanged());
}
@@ -235,14 +264,18 @@ public class AzureMetadataTypeEditor extends
MetadataEditor<AzureMetadataType> {
wStorageAccountName.setText(Const.NVL(azureMetadataType.getStorageAccountName(),
""));
wAuthenticationType.setText(Const.NVL(azureMetadataType.getAuthenticationType(),
"Key"));
wStorageAccountKey.setText(Const.NVL(azureMetadataType.getStorageAccountKey(),
""));
+ wSasToken.setText(Const.NVL(azureMetadataType.getSasToken(), ""));
wStorageAccountEndpoint.setText(Const.NVL(azureMetadataType.getStorageAccountEndpoint(),
""));
wCacheTtlSeconds.setText(Const.NVL(azureMetadataType.getCacheTtlSeconds(),
"5"));
// Show/hide storage account key based on authentication type
String authType = wAuthenticationType.getText();
boolean showKey = "Key".equals(authType);
+ boolean showSasToken = "SAS Token".equals(authType);
wlStorageAccountKey.setVisible(showKey);
wStorageAccountKey.setVisible(showKey);
+ wlSasToken.setVisible(showSasToken);
+ wSasToken.setVisible(showSasToken);
}
@Override
@@ -252,6 +285,7 @@ public class AzureMetadataTypeEditor extends
MetadataEditor<AzureMetadataType> {
azureMetadataType.setStorageAccountName(wStorageAccountName.getText());
azureMetadataType.setAuthenticationType(wAuthenticationType.getText());
azureMetadataType.setStorageAccountKey(wStorageAccountKey.getText());
+ azureMetadataType.setSasToken(wSasToken.getText());
azureMetadataType.setStorageAccountEndpoint(wStorageAccountEndpoint.getText());
azureMetadataType.setCacheTtlSeconds(wCacheTtlSeconds.getText());
}
diff --git
a/plugins/tech/azure/src/main/resources/org/apache/hop/vfs/azure/metadatatype/messages/messages_en_US.properties
b/plugins/tech/azure/src/main/resources/org/apache/hop/vfs/azure/metadatatype/messages/messages_en_US.properties
index dc56684b92..b01ddd3247 100644
---
a/plugins/tech/azure/src/main/resources/org/apache/hop/vfs/azure/metadatatype/messages/messages_en_US.properties
+++
b/plugins/tech/azure/src/main/resources/org/apache/hop/vfs/azure/metadatatype/messages/messages_en_US.properties
@@ -24,6 +24,8 @@ AzureMetadataTypeEditor.Description.Label=Description
AzureMetadataTypeEditor.StorageAccountName.Label=Storage Account Name
AzureMetadataTypeEditor.AuthenticationType.Label=Authentication Type
AzureMetadataTypeEditor.StorageAccountKey.Label=Storage Account Key
+AzureMetadataTypeEditor.SasToken.Label=SAS Token
+AzureMetadataTypeEditor.SasToken.ToolTip=A shared access signature, as the
query string only. The leading '?' is optional. Do not paste a full URL.
AzureMetadataTypeEditor.StorageAccountEndpoint.Label=Storage Endpoint
AzureMetadataTypeEditor.CacheTtlSeconds.Label=Cache TTL (seconds)
AzureMetadataTypeEditor.CacheTtlSeconds.Description=How long to cache folder
listing results (in seconds, default 5). If not set, falls back to 10 seconds.
diff --git
a/plugins/tech/azure/src/test/java/org/apache/hop/vfs/azure/AzureSasTokenAuthenticationTest.java
b/plugins/tech/azure/src/test/java/org/apache/hop/vfs/azure/AzureSasTokenAuthenticationTest.java
new file mode 100644
index 0000000000..48684f4afe
--- /dev/null
+++
b/plugins/tech/azure/src/test/java/org/apache/hop/vfs/azure/AzureSasTokenAuthenticationTest.java
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.vfs.azure;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.lang.reflect.Field;
+import org.apache.hop.metadata.api.HopMetadataProperty;
+import org.apache.hop.vfs.azure.metadatatype.AzureMetadataType;
+import org.junit.jupiter.api.Test;
+
+/** Tests for the SAS token authentication option of the Azure VFS plugin. */
+class AzureSasTokenAuthenticationTest {
+
+ @Test
+ void sasTokenIsStoredOnTheConnectionMetadata() {
+ AzureMetadataType metadataType = new AzureMetadataType();
+ metadataType.setAuthenticationType("SAS Token");
+
metadataType.setSasToken("sv=2022-11-02&ss=b&srt=co&sp=rl&se=2026-12-31T00:00:00Z&sig=abc123");
+
+ assertEquals("SAS Token", metadataType.getAuthenticationType());
+ assertEquals(
+ "sv=2022-11-02&ss=b&srt=co&sp=rl&se=2026-12-31T00:00:00Z&sig=abc123",
+ metadataType.getSasToken());
+ }
+
+ /**
+ * A SAS token grants access on its own, so it has to be treated as a secret
exactly like the
+ * storage account key: {@code password = true} is what makes Hop encrypt it
at rest and mask it
+ * in the UI. The whole point of the option is to avoid handing out long
lived credentials, which
+ * is undone if the token is written to metadata in clear text.
+ */
+ @Test
+ void sasTokenIsMarkedAsAPassword() throws NoSuchFieldException {
+ Field sasToken = AzureMetadataType.class.getDeclaredField("sasToken");
+ HopMetadataProperty property =
sasToken.getAnnotation(HopMetadataProperty.class);
+
+ assertNotNull(property, "sasToken must be a HopMetadataProperty to be
serialized");
+ assertTrue(property.password(), "sasToken must be marked as a password so
it is encrypted");
+ }
+
+ /** The storage account key must keep the same protection, so the two stay
consistent. */
+ @Test
+ void storageAccountKeyRemainsAPassword() throws NoSuchFieldException {
+ Field key = AzureMetadataType.class.getDeclaredField("storageAccountKey");
+ HopMetadataProperty property =
key.getAnnotation(HopMetadataProperty.class);
+
+ assertNotNull(property);
+ assertTrue(property.password());
+ }
+
+ /**
+ * Adding a third authentication type must not disturb existing connections.
A new instance still
+ * defaults to "Key", and a key-based connection carries no SAS token, so
nothing an existing user
+ * has configured changes meaning.
+ */
+ @Test
+ void keyAuthenticationRemainsTheDefault() {
+ AzureMetadataType metadataType = new AzureMetadataType();
+ metadataType.setStorageAccountName("hopsa");
+ metadataType.setStorageAccountKey("aGVsbG93b3JsZA==");
+
+ assertEquals("Key", metadataType.getAuthenticationType());
+ assertNull(metadataType.getSasToken());
+ }
+}
