[
https://issues.apache.org/jira/browse/HADOOP-19379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17910006#comment-17910006
]
ASF GitHub Bot commented on HADOOP-19379:
-----------------------------------------
horizonzy commented on PR #7252:
URL: https://github.com/apache/hadoop/pull/7252#issuecomment-2572337108
Thanks @anujmodi2021
> Why was this change made on wasb driver? [Wasb is on a path of
deprecation](https://issues.apache.org/jira/browse/HADOOP-19178) and we already
support
The Azurite only supported blob protocol, not supported lake data protocol.
Refer: https://github.com/Azure/Azurite/issues/2520#issuecomment-2556314402.
> Why do we need any other URL when Azurite is supposed to run on localhost
always?
In our integration test case. We use the test container to start Azurite as
the storage service and start another container as the storage visitor. So the
storage visitor can't use localhost to access the Azurite service.
> What does "another container" means here? Do you want to work with
multiple filesystems?
It means the docker container as above.
> [ABFS] Support Azurite storage emulator
> ---------------------------------------
>
> Key: HADOOP-19379
> URL: https://issues.apache.org/jira/browse/HADOOP-19379
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/azure, test
> Affects Versions: 3.4.1
> Reporter: Yan Zhao
> Priority: Minor
> Labels: pull-request-available
>
> In the integration test case, we will start the Azurite using the test
> container, and we need to access Azurite at another container.
> Now, the Azurite emulator uri values is fixed at 127.0.0.1, so at another
> container, it can't access the Azurite service.
> So I want to introduce a new config `fs.azure.storage.emulator.proxy.url`, so
> that I can config `fs.azure.storage.emulator.proxy.url=http://\{AzuriteIp}`,
> then it can access the Azurite service.
>
>
> {code:java}
> private void connectUsingCredentials(String accountName,
> StorageCredentials credentials, String containerName)
> throws URISyntaxException, StorageException, AzureException {
> URI blobEndPoint;
> if (isStorageEmulatorAccount(accountName)) {
> isStorageEmulator = true;
> CloudStorageAccount account =
> CloudStorageAccount.getDevelopmentStorageAccount();
> storageInteractionLayer.createBlobClient(account);
> } else {
> blobEndPoint = new URI(getHTTPScheme() + "://" + accountName);
> storageInteractionLayer.createBlobClient(blobEndPoint, credentials);
> }
> suppressRetryPolicyInClientIfNeeded();
> // Capture the container reference for debugging purposes.
> container = storageInteractionLayer.getContainerReference(containerName);
> rootDirectory = container.getDirectoryReference("");
> // Can only create container if using account key credentials
> canCreateOrModifyContainer = credentials instanceof
> StorageCredentialsAccountAndKey;
> } {code}
>
> {code:java}
> public static CloudStorageAccount getDevelopmentStorageAccount() {
> try {
> return getDevelopmentStorageAccount(null);
> }
> catch (final URISyntaxException e) {
> // this won't happen since we know the standard development stororage
> uri is valid.
> return null;
> }
> }
> public static CloudStorageAccount getDevelopmentStorageAccount(final URI
> proxyUri) throws URISyntaxException {
> String scheme;
> String host;
> if (proxyUri == null) {
> scheme = "http";
> host = "127.0.0.1";
> }
> else {
> scheme = proxyUri.getScheme();
> host = proxyUri.getHost();
> }
> StorageCredentials credentials = new
> StorageCredentialsAccountAndKey(DEVSTORE_ACCOUNT_NAME,
> DEVSTORE_ACCOUNT_KEY);
> URI blobPrimaryEndpoint = new
> URI(String.format(DEVELOPMENT_STORAGE_PRIMARY_ENDPOINT_FORMAT, scheme, host,
> "10000", DEVSTORE_ACCOUNT_NAME));
> URI queuePrimaryEndpoint = new
> URI(String.format(DEVELOPMENT_STORAGE_PRIMARY_ENDPOINT_FORMAT, scheme, host,
> "10001", DEVSTORE_ACCOUNT_NAME));
> URI tablePrimaryEndpoint = new
> URI(String.format(DEVELOPMENT_STORAGE_PRIMARY_ENDPOINT_FORMAT, scheme, host,
> "10002", DEVSTORE_ACCOUNT_NAME));
> URI blobSecondaryEndpoint = new
> URI(String.format(DEVELOPMENT_STORAGE_SECONDARY_ENDPOINT_FORMAT, scheme, host,
> "10000", DEVSTORE_ACCOUNT_NAME));
> URI queueSecondaryEndpoint = new
> URI(String.format(DEVELOPMENT_STORAGE_SECONDARY_ENDPOINT_FORMAT, scheme, host,
> "10001", DEVSTORE_ACCOUNT_NAME));
> URI tableSecondaryEndpoint = new
> URI(String.format(DEVELOPMENT_STORAGE_SECONDARY_ENDPOINT_FORMAT, scheme, host,
> "10002", DEVSTORE_ACCOUNT_NAME));
> CloudStorageAccount account = new CloudStorageAccount(credentials, new
> StorageUri(blobPrimaryEndpoint,
> blobSecondaryEndpoint), new StorageUri(queuePrimaryEndpoint,
> queueSecondaryEndpoint), new StorageUri(
> tablePrimaryEndpoint, tableSecondaryEndpoint), null /*
> fileStorageUri */);
> account.isDevStoreAccount = true;
> return account;
> } {code}
> CloudStorageAccount account =
> CloudStorageAccount.getDevelopmentStorageAccount(); will using 127.0.0.1 as
> the `azurite` host.
>
> In fact, here we can pass into a proxy uri by invoke
> getDevelopmentStorageAccount(final URI proxyUri)
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
