josh-fell edited a comment on issue #19409:
URL: https://github.com/apache/airflow/issues/19409#issuecomment-985975109
When I test this using `main` specifically with a container-level SAS token
with all available perms on said container, the following exception is thrown:
```
[2021-12-03, 23:17:34 EST] {taskinstance.py:1698} ERROR - Task failed with
exception
Traceback (most recent call last):
File
"/usr/local/lib/python3.7/site-packages/azure/storage/blob/_container_client.py",
line 298, in create_container
**kwargs)
File
"/usr/local/lib/python3.7/site-packages/azure/storage/blob/_generated/operations/_container_operations.py",
line 134, in create
raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: Operation returned an invalid
status 'This request is not authorized to perform this operation.'
```
_Digging, part 1_
Currently when attempting to upload a blob, we are attempting to first
create a container, if it already exists, simply return the `ContainerClient`
to continue the upload. However, only a `ResourceExistsError` is caught during
the creation step which leads to task failure.
_Digging, part 2_
Looking at the [documentation for service SAS
perms](https://docs.microsoft.com/en-us/rest/api/storageservices/create-service-sas#permissions-for-a-directory-container-or-blob)
re: containers/blobs:
>A service SAS cannot grant access to certain operations:
>
> - Containers, queues, and tables cannot be created, deleted, or listed.
> - Container metadata and properties cannot be read or written.
> ...
> - Containers cannot be leased.
>
>To construct a SAS that grants access to these operations, use an account
SAS.
Reading container properties is part of creating containers which again
leads to the above exception.
There is also a discussion
[here](https://github.com/Azure/azure-sdk-for-python/issues/18012) about this
too.
_Possible options/next steps_
1. We can continue to inherently disallow the use of container-level SAS to
upload blobs without updating any logic in the provider and document this
constraint. This of course assumes that the behavior to _always_ attempt
creating a container is desired.
2. Check if a SAS Token is used to authenticate and if it is a
container-level SAS (meaning `sr=c` is in the URI). If yes, do not create a
container, else proceed with the attempt to create a container per status quo.
3. Simply catch a `HttpResponseError` exception when attempting to perform
the container creation. Again, keeping the assumption that container creation
attempts are desired here.
4. Refactor the blob `upload` mechanism in the provider to make the
container creation configurable.
P.S. Generally speaking the `WasbHook` methods seem to favor using a
`ContainerClient` to perform blob-level operations even though this hook
specifically interacts with Azure Blob Storage using a `BlobServiceClient`
which can directly perform these operations. Presumably this exists because of
the "auto-container creation" behavior. IMO this behavior seems worthy of
revisiting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
