[jira] [Comment Edited] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage
[
https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998110#comment-16998110
]
Manju edited comment on JCLOUDS-1428 at 12/17/19 12:14 PM:
---
*Jcloud version - 2.2.0*
*Using Azure Secret Key: Working fine*
{code:java}
it should "Upload a file" in {
val sasToken = AppConf.getSSAToken("azure")
val azureKey = AppConf.getStorageKey("azure")
val azureToken = AppConf.getStorageSecret("azure")
import org.jclouds.ContextBuilder
import org.jclouds.blobstore.BlobStoreContext
val context = ContextBuilder.newBuilder("azureblob").credentials(azureKey,
azureToken).buildView(classOf[BlobStoreContext])
var blobStore = context.getBlobStore()
blobStore.createContainerInLocation(null, "mycontainer")
val fileObj = new File("src/test/resources/test-data.log")
val payload = Files.asByteSource(fileObj)
val blob =
blobStore.blobBuilder("test1/5453").payload(payload).contentLength(payload.size()).build()
blobStore.putBlob("mycontainer", blob, new PutOptions().multipart())
context.close()
}
{code}
*Using Azure SAS Token: It fails to upload*
{code:java}
it should "Upload a file" in {
val sasToken = AppConf.getSSAToken("azure")
val azureKey = AppConf.getStorageKey("azure")
val azureToken = AppConf.getStorageSecret("azure")
import org.jclouds.ContextBuilder
import org.jclouds.blobstore.BlobStoreContext
val context = ContextBuilder.newBuilder("azureblob").credentials(azureKey,
sasToken).buildView(classOf[BlobStoreContext])
var blobStore = context.getBlobStore()
blobStore.createContainerInLocation(null, "mycontainer")
val fileObj = new File("src/test/resources/test-data.log")
val payload = Files.asByteSource(fileObj)
val blob =
blobStore.blobBuilder("test1/5453").payload(payload).contentLength(payload.size()).build()
blobStore.putBlob("mycontainer", blob, new PutOptions().multipart())
context.close()
}
{code}
*Error: When using SAS Token*
{code:java}
org.jclouds.azure.storage.AzureStorageResponseException: command
[method=org.jclouds.azureblob.AzureBlobClient.public abstract void
org.jclouds.azureblob.AzureBlobClient.putBlock(java.lang.String,java.lang.String,java.lang.String,org.jclouds.io.Payload)[mycontainer,
test1/5453, AQ==, [content=true, contentMetadata=[cacheControl=null,
contentDisposition=null, contentEncoding=null, contentLanguage=null,
contentLength=7986, contentMD5=null, contentType=application/unknown,
expires=null], written=false, isSensitive=false]], request=PUT
https://test.blob.core.windows.net/mycontainer/test1/5453?comp=block=AQ%3D%3D
HTTP/1.1] failed with code 400, error:
AzureError{requestId='c13bc6b2-f01e-0020-5acd-b4114e00',
code='InvalidQueryParameterValue', message='Value for one of the query
parameters specified in the request URI is invalid.
RequestId:c13bc6b2-f01e-0020-5acd-b4114e00
Time:2019-12-17T11:31:17.8460459Z', context='{QueryParameterValue=block,
QueryParameterName=comp, Reason=}'}
com.google.common.util.concurrent.UncheckedExecutionException:
org.jclouds.azure.storage.AzureStorageResponseException: command
[method=org.jclouds.azureblob.AzureBlobClient.public abstract void
org.jclouds.azureblob.AzureBlobClient.putBlock(java.lang.String,java.lang.String,java.lang.String,org.jclouds.io.Payload)[mycontainer,
test1/5453, AQ==, [content=true, contentMetadata=[cacheControl=null,
contentDisposition=null, contentEncoding=null, contentLanguage=null,
contentLength=7986, contentMD5=null, contentType=application/unknown,
expires=null], written=false, isSensitive=false]], request=PUT
https://test.blob.core.windows.net/mycontainer/test1/5453?comp=block=AQ%3D%3D
HTTP/1.1] failed with code 400, error:
AzureError{requestId='c13bc6b2-f01e-0020-5acd-b4114e00',
code='InvalidQueryParameterValue', message='Value for one of the query
parameters specified in the request URI is invalid.
RequestId:c13bc6b2-f01e-0020-5acd-b4114e00
Time:2019-12-17T11:31:17.8460459Z', context='{QueryParameterValue=block,
QueryParameterName=comp, Reason=}'}
{code}
*Observation:* I think with SAS token *blobStore.blobBuilder("test1/5453")*
when we use a path like this format(test1/5453) then it's breaking.
It works when i just use *blobStore.blobBuilder("test1")*
[~gaul] [~himanshujain] [~jira-bot]
was (Author: manjunathdavanam):
*Jcloud version - 2.2.0*
*Using Azure Secret Key: Working fine*
{code:java}
it should "Upload a file" in {
val sasToken = AppConf.getSSAToken("azure")
val azureKey = AppConf.getStorageKey("azure")
val azureToken = AppConf.getStorageSecret("azure")
import org.jclouds.ContextBuilder
import org.jclouds.blobstore.BlobStoreContext
val context = ContextBuilder.newBuilder("azureblob").credentials(azureKey,
[jira] [Comment Edited] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage
[
https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998110#comment-16998110
]
Manju edited comment on JCLOUDS-1428 at 12/17/19 11:37 AM:
---
*Jcloud version - 2.2.0*
*Using Azure Secret Key: Working fine*
{code:java}
it should "Upload a file" in {
val sasToken = AppConf.getSSAToken("azure")
val azureKey = AppConf.getStorageKey("azure")
val azureToken = AppConf.getStorageSecret("azure")
import org.jclouds.ContextBuilder
import org.jclouds.blobstore.BlobStoreContext
val context = ContextBuilder.newBuilder("azureblob").credentials(azureKey,
azureToken).buildView(classOf[BlobStoreContext])
var blobStore = context.getBlobStore()
blobStore.createContainerInLocation(null, "mycontainer")
val fileObj = new File("src/test/resources/test-data.log")
val payload = Files.asByteSource(fileObj)
val blob =
blobStore.blobBuilder("test1/5453").payload(payload).contentLength(payload.size()).build()
blobStore.putBlob("mycontainer", blob, new PutOptions().multipart())
context.close()
}
{code}
*Using Azure SAS Token: It fails to upload*
{code:java}
it should "Upload a file" in {
val sasToken = AppConf.getSSAToken("azure")
val azureKey = AppConf.getStorageKey("azure")
val azureToken = AppConf.getStorageSecret("azure")
import org.jclouds.ContextBuilder
import org.jclouds.blobstore.BlobStoreContext
val context = ContextBuilder.newBuilder("azureblob").credentials(azureKey,
sasToken).buildView(classOf[BlobStoreContext])
var blobStore = context.getBlobStore()
blobStore.createContainerInLocation(null, "mycontainer")
val fileObj = new File("src/test/resources/test-data.log")
val payload = Files.asByteSource(fileObj)
val blob =
blobStore.blobBuilder("test1/5453").payload(payload).contentLength(payload.size()).build()
blobStore.putBlob("mycontainer", blob, new PutOptions().multipart())
context.close()
}
{code}
*Error: When using SAS Token*
{code:java}
org.jclouds.azure.storage.AzureStorageResponseException: command
[method=org.jclouds.azureblob.AzureBlobClient.public abstract void
org.jclouds.azureblob.AzureBlobClient.putBlock(java.lang.String,java.lang.String,java.lang.String,org.jclouds.io.Payload)[mycontainer,
test1/5453, AQ==, [content=true, contentMetadata=[cacheControl=null,
contentDisposition=null, contentEncoding=null, contentLanguage=null,
contentLength=7986, contentMD5=null, contentType=application/unknown,
expires=null], written=false, isSensitive=false]], request=PUT
https://test.blob.core.windows.net/mycontainer/test1/5453?comp=block=AQ%3D%3D
HTTP/1.1] failed with code 400, error:
AzureError{requestId='c13bc6b2-f01e-0020-5acd-b4114e00',
code='InvalidQueryParameterValue', message='Value for one of the query
parameters specified in the request URI is invalid.
RequestId:c13bc6b2-f01e-0020-5acd-b4114e00
Time:2019-12-17T11:31:17.8460459Z', context='{QueryParameterValue=block,
QueryParameterName=comp, Reason=}'}
com.google.common.util.concurrent.UncheckedExecutionException:
org.jclouds.azure.storage.AzureStorageResponseException: command
[method=org.jclouds.azureblob.AzureBlobClient.public abstract void
org.jclouds.azureblob.AzureBlobClient.putBlock(java.lang.String,java.lang.String,java.lang.String,org.jclouds.io.Payload)[mycontainer,
test1/5453, AQ==, [content=true, contentMetadata=[cacheControl=null,
contentDisposition=null, contentEncoding=null, contentLanguage=null,
contentLength=7986, contentMD5=null, contentType=application/unknown,
expires=null], written=false, isSensitive=false]], request=PUT
https://test.blob.core.windows.net/mycontainer/test1/5453?comp=block=AQ%3D%3D
HTTP/1.1] failed with code 400, error:
AzureError{requestId='c13bc6b2-f01e-0020-5acd-b4114e00',
code='InvalidQueryParameterValue', message='Value for one of the query
parameters specified in the request URI is invalid.
RequestId:c13bc6b2-f01e-0020-5acd-b4114e00
Time:2019-12-17T11:31:17.8460459Z', context='{QueryParameterValue=block,
QueryParameterName=comp, Reason=}'}
{code}
*Observation:* I think with SAS token *blobStore.blobBuilder("test1/5453")*
when we use a path like this format(test1/5453) then it's breaking.
It works when i just use *blobStore.blobBuilder("test1/5453")*
[~gaul] [~himanshujain] [~jira-bot]
was (Author: manjunathdavanam):
*Using Azure Secret Key: Working fine*
{code:java}
it should "Upload a file" in {
val sasToken = AppConf.getSSAToken("azure")
val azureKey = AppConf.getStorageKey("azure")
val azureToken = AppConf.getStorageSecret("azure")
import org.jclouds.ContextBuilder
import org.jclouds.blobstore.BlobStoreContext
val context = ContextBuilder.newBuilder("azureblob").credentials(azureKey,
[jira] [Comment Edited] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage
[
https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998029#comment-16998029
]
Manju edited comment on JCLOUDS-1428 at 12/17/19 9:55 AM:
--
[~gaul] [~himanshujain]
Any example of how to use the SAS token? with the below snippet we are getting
400 error code while uploading the file to the container.
{code:java}
Caused by: org.jclouds.azure.storage.AzureStorageResponseException: command
[method=org.jclouds.azureblob.AzureBlobClient.public abstract void
org.jclouds.azureblob.AzureBlobClient.putBlock(java.lang.String,java.lang.String,java.lang.String,org.jclouds.io.Payload)[test-container,
testUpload/test-blob.log, AQ==, [content=true,
contentMetadata=[cacheControl=null, contentDisposition=null,
contentEncoding=UTF-8, contentLanguage=null, contentLength=7986,
contentMD5=null, contentType=text/x-log, expires=null], written=false,
isSensitive=false]], request=PUT
https://XXX/test-container/testUpload/test-blob.log?comp=block=AQ%3D%3D
HTTP/1.1] failed with code 400, error:
AzureError{requestId='737e1a19-001e-0017-20bd-b4bde100',
code='InvalidQueryParameterValue', message='Value for one of the query
parameters specified in the request URI is invalid.
RequestId:737e1a19-001e-0017-20bd-b4bde100
Time:2019-12-17T09:38:40.3385715Z', context='{QueryParameterValue=block,
QueryParameterName=comp, Reason=}'}
{code}
{code:java}
BlobStoreContext context = ContextBuilder.newBuilder("azureblob")
.credentials(storageAccountName, sasToken)
.buildView(BlobStoreContext.class);
{code}
was (Author: manjunathdavanam):
[~gaul] [~himanshujain]
Any example of how to use the SAS token? with the below snippet we are getting
400 error code while uploading the file to the container.
{code:java}
failed with code 400, error:
AzureError{requestId='737e1a19-001e-0017-20bd-b4bde100',
code='InvalidQueryParameterValue', message='Value for one of the query
parameters specified in the request URI is invalid.
{code}
{code:java}
BlobStoreContext context = ContextBuilder.newBuilder("azureblob")
.credentials(storageAccountName, sasToken)
.buildView(BlobStoreContext.class);
{code}
> Support for SAS token based Authentication for Azure Blob Storage
> -
>
> Key: JCLOUDS-1428
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1428
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-blobstore
>Reporter: Himanshu Jain
>Assignee: Andrew Gaul
>Priority: Major
> Labels: azureblob
> Fix For: 2.2.0, 2.1.3
>
> Attachments: azure_stacktrace.txt
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Hi,
> We have one use case where we want to provide limited access to objects in
> our storage accounts. We figured that the best way to do this is by using
> SAS token based authentication mechanism to upload/download objects to Azure
> Blob Storage - [SAS based
> Authentication|https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1]
> We found that JClouds client library provides support for Azure Blob Storage
> using account keys which might not fit our use case because of security
> reasons.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage
[
https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998029#comment-16998029
]
Manju edited comment on JCLOUDS-1428 at 12/17/19 9:53 AM:
--
[~gaul] [~himanshujain]
Any example of how to use the SAS token? with the below snippet we are getting
400 error code while uploading the file to the container.
{code:java}
failed with code 400, error:
AzureError{requestId='737e1a19-001e-0017-20bd-b4bde100',
code='InvalidQueryParameterValue', message='Value for one of the query
parameters specified in the request URI is invalid.
{code}
{code:java}
BlobStoreContext context = ContextBuilder.newBuilder("azureblob")
.credentials(storageAccountName, sasToken)
.buildView(BlobStoreContext.class);
{code}
was (Author: manjunathdavanam):
[~gaul] [~himanshujain]
Any example of how to use the SAS token? with the below snippet we are getting
400 error code while uploading the file to the container.
{code:java}
failed with code 400, error:
AzureError{requestId='737e1a19-001e-0017-20bd-b4bde100',
code='InvalidQueryParameterValue', message='Value for one of the query
parameters specified in the request URI is invalid.
{code}
{code:java}
BlobStoreContext context = ContextBuilder.newBuilder("azureblob") //using
sasToken in place of storage key .credentials(storageAccountName, sasToken)
.buildView(BlobStoreContext.class);
{code}
> Support for SAS token based Authentication for Azure Blob Storage
> -
>
> Key: JCLOUDS-1428
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1428
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-blobstore
>Reporter: Himanshu Jain
>Assignee: Andrew Gaul
>Priority: Major
> Labels: azureblob
> Fix For: 2.2.0, 2.1.3
>
> Attachments: azure_stacktrace.txt
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Hi,
> We have one use case where we want to provide limited access to objects in
> our storage accounts. We figured that the best way to do this is by using
> SAS token based authentication mechanism to upload/download objects to Azure
> Blob Storage - [SAS based
> Authentication|https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1]
> We found that JClouds client library provides support for Azure Blob Storage
> using account keys which might not fit our use case because of security
> reasons.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage
[
https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16998029#comment-16998029
]
Manju edited comment on JCLOUDS-1428 at 12/17/19 9:52 AM:
--
[~gaul] [~himanshujain]
Any example of how to use the SAS token? with the below snippet we are getting
400 error code while uploading the file to the container.
{code:java}
failed with code 400, error:
AzureError{requestId='737e1a19-001e-0017-20bd-b4bde100',
code='InvalidQueryParameterValue', message='Value for one of the query
parameters specified in the request URI is invalid.
{code}
{code:java}
BlobStoreContext context = ContextBuilder.newBuilder("azureblob") //using
sasToken in place of storage key .credentials(storageAccountName, sasToken)
.buildView(BlobStoreContext.class);
{code}
was (Author: manjunathdavanam):
[~gaul] [~himanshujain]
Any example of how to use the SAS token? with the below snippet we are getting
400 error code while uploading the file to the container.
{code:java}
failed with code 400, error:
AzureError{requestId='737e1a19-001e-0017-20bd-b4bde100',
code='InvalidQueryParameterValue', message='Value for one of the query
parameters specified in the request URI is invalid.
{code}
BlobStoreContext context = ContextBuilder.newBuilder("azureblob") //using
sasToken in place of storage key .credentials(storageAccountName, sasToken)
.buildView(BlobStoreContext.class);
> Support for SAS token based Authentication for Azure Blob Storage
> -
>
> Key: JCLOUDS-1428
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1428
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-blobstore
>Reporter: Himanshu Jain
>Assignee: Andrew Gaul
>Priority: Major
> Labels: azureblob
> Fix For: 2.2.0, 2.1.3
>
> Attachments: azure_stacktrace.txt
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Hi,
> We have one use case where we want to provide limited access to objects in
> our storage accounts. We figured that the best way to do this is by using
> SAS token based authentication mechanism to upload/download objects to Azure
> Blob Storage - [SAS based
> Authentication|https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1]
> We found that JClouds client library provides support for Azure Blob Storage
> using account keys which might not fit our use case because of security
> reasons.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage
[ https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16823741#comment-16823741 ] Biswa Ranjan Ray edited comment on JCLOUDS-1428 at 4/24/19 4:23 AM: Hi Folks, Would really appreciate if you could provide any update on this issue. Thanks. was (Author: roy.biswa): Hi Folks, Would really appreciate if could provide any update on this issue. Thanks. > Support for SAS token based Authentication for Azure Blob Storage > - > > Key: JCLOUDS-1428 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1428 > Project: jclouds > Issue Type: Improvement > Components: jclouds-blobstore >Reporter: Himanshu Jain >Priority: Major > Labels: azureblob > Fix For: 2.2.0, 2.1.3 > > Attachments: azure_stacktrace.txt > > Time Spent: 40m > Remaining Estimate: 0h > > Hi, > We have one use case where we want to provide limited access to objects in > our storage accounts. We figured that the best way to do this is by using > SAS token based authentication mechanism to upload/download objects to Azure > Blob Storage - [SAS based > Authentication|https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1] > We found that JClouds client library provides support for Azure Blob Storage > using account keys which might not fit our use case because of security > reasons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage
[ https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824802#comment-16824802 ] Biswa Ranjan Ray edited comment on JCLOUDS-1428 at 4/24/19 4:22 AM: [~Horuszko] Thank you for your update. was (Author: roy.biswa): [~Horuszko] Thank you for the update. I'll try out the pull request. > Support for SAS token based Authentication for Azure Blob Storage > - > > Key: JCLOUDS-1428 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1428 > Project: jclouds > Issue Type: Improvement > Components: jclouds-blobstore >Reporter: Himanshu Jain >Priority: Major > Labels: azureblob > Fix For: 2.2.0, 2.1.3 > > Attachments: azure_stacktrace.txt > > Time Spent: 40m > Remaining Estimate: 0h > > Hi, > We have one use case where we want to provide limited access to objects in > our storage accounts. We figured that the best way to do this is by using > SAS token based authentication mechanism to upload/download objects to Azure > Blob Storage - [SAS based > Authentication|https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1] > We found that JClouds client library provides support for Azure Blob Storage > using account keys which might not fit our use case because of security > reasons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage
[
https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806226#comment-16806226
]
Ignasi Barrera edited comment on JCLOUDS-1428 at 3/31/19 6:15 PM:
--
It is called
[here|https://github.com/apache/jclouds/blob/master/providers/azureblob/src/main/java/org/jclouds/azureblob/blobstore/functions/BlobPropertiesToBlobMetadata.java#L58].
That {{BlobPropertiesToBlobMetadata}} function is called everytime a blob is
retrieved using the {{AzureBlobStore}}. If you look at the usages of [these
two|https://github.com/apache/jclouds/blob/master/providers/azureblob/src/main/java/org/jclouds/azureblob/blobstore/AzureBlobStore.java#L93-L94]
variables and [this
one|https://github.com/apache/jclouds/blob/master/providers/azureblob/src/main/java/org/jclouds/azureblob/blobstore/AzureBlobStore.java#L96],
you'll see when that gets called, which is basically when getting a blob, or
getting a list of blobs, so, in practice, it will always fail for SAS tokens,
I wonder if it would make sense to disable getting that info when we already
know it can't be retrieved with the provided credentials? [~gaul] WDYT?
was (Author: nacx):
It is called
[here|https://github.com/apache/jclouds/blob/master/providers/azureblob/src/main/java/org/jclouds/azureblob/blobstore/functions/BlobPropertiesToBlobMetadata.java#L58].
That {{BlobPropertiesToBlobMetadata}} function is called everytime a blob is
retrieved using the {{AzureBlobStore}}. If you look at the usages of [these
two|https://github.com/apache/jclouds/blob/master/providers/azureblob/src/main/java/org/jclouds/azureblob/blobstore/AzureBlobStore.java#L93-L94]
variables and [this
one|https://github.com/apache/jclouds/blob/master/providers/azureblob/src/main/java/org/jclouds/azureblob/blobstore/AzureBlobStore.java#L96],
you'll see when that gets called, which is basically when getting a blob, or
getting a list of blobs, so, in practice, it will always fail for SAS tokens,
I wonder if it would make sense to disable getting that info when we already
know it can't be retrieved with the provided credntials? [~gaul] WDYT?
> Support for SAS token based Authentication for Azure Blob Storage
> -
>
> Key: JCLOUDS-1428
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1428
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-blobstore
>Reporter: Himanshu Jain
>Priority: Major
> Labels: azureblob
> Fix For: 2.2.0, 2.1.3
>
> Attachments: azure_stacktrace.txt
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Hi,
> We have one use case where we want to provide limited access to objects in
> our storage accounts. We figured that the best way to do this is by using
> SAS token based authentication mechanism to upload/download objects to Azure
> Blob Storage - [SAS based
> Authentication|https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1]
> We found that JClouds client library provides support for Azure Blob Storage
> using account keys which might not fit our use case because of security
> reasons.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage
[
https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791408#comment-16791408
]
Swati Jain edited comment on JCLOUDS-1428 at 3/13/19 7:49 AM:
--
Hi [~Horuszko]
Based on the discussion above and the understanding that we have so far, these
4 different keys are assumed to be mandatory "sig", "se", "sv", "sp" for a SAS
token in current implementation of JClouds.
We would want to let you know the following, (may be you already know about it)
:
* There are 2 types of SAS tokens - 1) Service SAS and 2) Account SAS. We make
use of *Service SAS*.
[Reference|[https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1#types-of-shared-access-signatures]]
* There are 2 forms of SAS tokens - 1) Ad hoc SAS and 2) SAS with stored
access policy. We make use of *SAS with stored access policy.*
[Reference|[https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1#controlling-a-sas-with-a-stored-access-policy]]
Please note that a SAS created using stored access policy inherits expiry time
and permissions from the policy itself. Azure documentation states the
following :
{quote}When you associate a SAS with a stored access policy, the SAS inherits
the constraints the start time, expiry time, and permissions, defined for the
stored access policy.
{quote}
As we make use of *Service SAS with stored access policy*, the "se" and "sp"
parameters aren't part of such a SAS token. Still such a token is valid and we
are able to make all the allowed operations on the given resource.
Also, please note that such a token is generated using the Azure SDK. Following
are some code snippets :
// Specifying permissions
{code:java}
EnumSet permissions =
EnumSet.of(SharedAccessBlobPermissions.READ,
SharedAccessBlobPermissions.ADD, SharedAccessBlobPermissions.CREATE,
SharedAccessBlobPermissions.WRITE,
SharedAccessBlobPermissions.DELETE, SharedAccessBlobPermissions.LIST);{code}
// Creating policy
{code:java}
SharedAccessBlobPolicy sharedAccessBlobPolicy = new
SharedAccessBlobPolicy();{code}
// Assigning expiry time and permissions to the policy
{code:java}
sharedAccessBlobPolicy.setPermissions(permissions);
sharedAccessBlobPolicy.setSharedAccessStartTime(startTime);
sharedAccessBlobPolicy.setSharedAccessExpiryTime(endTime);
{code}
// Generating SAS token using the above policy
[Reference|[https://azure.github.io/azure-sdk-for-java/com/microsoft/azure/storage/blob/CloudBlobContainer.html#generateSharedAccessSignature-com.microsoft.azure.storage.blob.SharedAccessBlobPolicy-java.lang.String-com.microsoft.azure.storage.IPRange-com.microsoft.azure.storage.SharedAccessProtocols-]]
{code:java}
String sasToken = cloudBlobContainer.generateSharedAccessSignature(null,
policyIdentifier, null, SharedAccessProtocols.HTTPS_ONLY);{code}
Using the above method, the SAS token generated doesn't contain "se" and "sp"
parameters. Rather it contains "si" parameter, that refers to signed identifier.
Therefore, the SAS token that we make use of is a valid one we believe.
Please let us know if the above explanation clarifies that the SAS token that
we generate and make use of is correct. If not, we can also have a short call
to discuss upon the same.
was (Author: swatijain1101):
Hi [~Horuszko]
Based on the discussion above and the understanding that we have so far, these
4 different keys are assumed to be mandatory "sig", "se", "sv", "sp" for a SAS
token in current implementation of JClouds.
We would want to let you know the following, (may be you already know about it)
:
* There are 2 types of SAS tokens - 1) Service SAS and 2) Account SAS. We make
use of *Service SAS*.
[Reference|[https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1#types-of-shared-access-signatures]]
* There are 2 forms of SAS tokens - 1) Ad hoc SAS and 2) SAS with stored
access policy. We make use of *SAS with stored access policy.*
[Reference|[https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1#controlling-a-sas-with-a-stored-access-policy]]
Please note that a SAS created using stored access policy inherits expiry time
and permissions from the policy itself. Azure documentation states the
following :
{quote}When you associate a SAS with a stored access policy, the SAS inherits
the constraints the start time, expiry time, and permissions, defined for the
stored access policy.
{quote}
As we make use of *Service SAS with stored access policy*, the "se" and "sp"
parameters aren't part of such a SAS token. Still such a token is valid and we
are able to make all the allowed operations on the given resource.
Also, please note that such a token is generated using the Azure SDK. Following
are some code snippets :
// Specifying
[jira] [Comment Edited] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage
[
https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791408#comment-16791408
]
Swati Jain edited comment on JCLOUDS-1428 at 3/13/19 7:51 AM:
--
Hi [~Horuszko]
Based on the discussion above and the understanding that we have so far, these
4 different keys are assumed to be mandatory "sig", "se", "sv", "sp" for a SAS
token in current implementation of JClouds.
We would want to let you know the following, (may be you already know about it)
:
* There are 2 types of SAS tokens - 1) Service SAS and 2) Account SAS. We make
use of *Service SAS*.
[Reference|[https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1#types-of-shared-access-signatures]]
* There are 2 forms of SAS tokens - 1) Ad hoc SAS and 2) SAS with stored
access policy. We make use of *SAS with stored access policy.*
[Reference|[https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1#controlling-a-sas-with-a-stored-access-policy]]
Please note that a SAS created using stored access policy inherits expiry time
and permissions from the policy itself. Azure documentation states the
following :
{quote}When you associate a SAS with a stored access policy, the SAS inherits
the constraints the start time, expiry time, and permissions, defined for the
stored access policy.
{quote}
As we make use of *Service SAS with stored access policy*, the "se" and "sp"
parameters aren't part of such a SAS token. Still such a token is valid and we
are able to make all the allowed operations on the given resource.
Also, please note that such a token is generated using the Azure SDK. Following
are some code snippets :
// Specifying permissions
{code:java}
EnumSet permissions =
EnumSet.of(SharedAccessBlobPermissions.READ,
SharedAccessBlobPermissions.ADD, SharedAccessBlobPermissions.CREATE,
SharedAccessBlobPermissions.WRITE,
SharedAccessBlobPermissions.DELETE, SharedAccessBlobPermissions.LIST);{code}
// Creating policy
{code:java}
SharedAccessBlobPolicy sharedAccessBlobPolicy = new
SharedAccessBlobPolicy();{code}
// Assigning expiry time and permissions to the policy
{code:java}
sharedAccessBlobPolicy.setPermissions(permissions);
sharedAccessBlobPolicy.setSharedAccessStartTime(startTime);
sharedAccessBlobPolicy.setSharedAccessExpiryTime(endTime);
{code}
// Generating SAS token using the above policy
[Reference|[https://azure.github.io/azure-sdk-for-java/com/microsoft/azure/storage/blob/CloudBlobContainer.html#generateSharedAccessSignature-com.microsoft.azure.storage.blob.SharedAccessBlobPolicy-java.lang.String-com.microsoft.azure.storage.IPRange-com.microsoft.azure.storage.SharedAccessProtocols-]]
{code:java}
String sasToken = cloudBlobContainer.generateSharedAccessSignature(null,
policyIdentifier, null, SharedAccessProtocols.HTTPS_ONLY);{code}
Using the above method, the SAS token generated doesn't contain "se" and "sp"
parameters. Rather it contains "si" parameter, that refers to signed identifier.
Therefore, the SAS token that we make use of is a valid one we believe.
Please let us know if the above explanation clarifies that the SAS token that
we generate and make use of is correct. If not, we can also have a short call
to discuss upon the same.
was (Author: swatijain1101):
Hi [~Horuszko]
Based on the discussion above and the understanding that we have so far, these
4 different keys are assumed to be mandatory "sig", "se", "sv", "sp" for a SAS
token in current implementation of JClouds.
We would want to let you know the following, (may be you already know about it)
:
* There are 2 types of SAS tokens - 1) Service SAS and 2) Account SAS. We make
use of *Service SAS*.
[Reference|[https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1#types-of-shared-access-signatures]]
* There are 2 forms of SAS tokens - 1) Ad hoc SAS and 2) SAS with stored
access policy. We make use of *SAS with stored access policy.*
[Reference|[https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1#controlling-a-sas-with-a-stored-access-policy]]
Please note that a SAS created using stored access policy inherits expiry time
and permissions from the policy itself. Azure documentation states the
following :
{quote}When you associate a SAS with a stored access policy, the SAS inherits
the constraints the start time, expiry time, and permissions, defined for the
stored access policy.
{quote}
As we make use of *Service SAS with stored access policy*, the "se" and "sp"
parameters aren't part of such a SAS token. Still such a token is valid and we
are able to make all the allowed operations on the given resource.
Also, please note that such a token is generated using the Azure SDK. Following
are some code snippets :
// Specifying
[jira] [Comment Edited] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage
[ https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16790281#comment-16790281 ] Alexandra Horuszko edited comment on JCLOUDS-1428 at 3/12/19 6:48 AM: -- [~roy.biswa] So, there is a check in the code, which defines, whether you use SAS or SharedKey in the following manner: it checks for the presence of four 'tokens': "sig", "se", "sv", "sp". In order to be classified as a SAS string, it needs to contain ALL tokens from this list. Your SAS string does not include "se" and "sp", that is why the code classifies it as a SharedKey, and then proceeds as if it was a SharedKey. And then it bumps into error, because it tries to decode it, as if it was a SharedKey, but it is actually not, it's too long, it contains wrong characters... Are you sure that such format of SAS will actually work on AzureBlobStorage? I can see that you're using Service SAS. And for Service SAS not only "sv" and "sig" are required, but also "se" and "sp". Please, take a look at the documentation here: [https://docs.microsoft.com/en-us/rest/api/storageservices/Constructing-a-Service-SAS?redirectedfrom=MSDN] . se - is the expiration timestamp, and sp is the permissions. was (Author: horuszko): [~roy.biswa] So, there is a check in the code, which defines, whether you use SAS or SharedKey in the following manner: it checks for the presence of four 'tokens': "sig", "se", "sv", "sp". In order to be classified as a SAS string, it needs to contain ALL tokens from this list. Your SAS string does not include "se" and "sp", that is why the code classifies it as a SharedKey, and then proceeds as if it was a SharedKey. And then it bumps into error, because it tries to decode it, as if it was a SharedKey, but it is actually not, it's too long, it contains wrong characters... Are you sure that such format of SAS will actually work on AzureBlobStorage? I can see that you're using Service SAS. And for Service SAS not only "sv" and "sig" are required, but also "se" and "sp". Please, take a look at the documentation here: [https://docs.microsoft.com/en-us/rest/api/storageservices/Constructing-a-Service-SAS?redirectedfrom=MSDN] . > Support for SAS token based Authentication for Azure Blob Storage > - > > Key: JCLOUDS-1428 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1428 > Project: jclouds > Issue Type: Improvement > Components: jclouds-blobstore >Reporter: Himanshu Jain >Priority: Major > Labels: azureblob > Fix For: 2.2.0, 2.1.3 > > Attachments: azure_stacktrace.txt > > > Hi, > We have one use case where we want to provide limited access to objects in > our storage accounts. We figured that the best way to do this is by using > SAS token based authentication mechanism to upload/download objects to Azure > Blob Storage - [SAS based > Authentication|https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1] > We found that JClouds client library provides support for Azure Blob Storage > using account keys which might not fit our use case because of security > reasons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage
[
https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16789302#comment-16789302
]
Biswa Ranjan Ray edited comment on JCLOUDS-1428 at 3/11/19 8:52 AM:
Hi,
I'm trying to configure BlobStoreContext for Azure storage using SAS token. I
found the below code snippet from [jclouds
|https://jclouds.apache.org/guides/azure-storage/]site that offers the portable
BlobStore api:
{code:java}
BlobStoreContext context = ContextBuilder.newBuilder("azureblob")
//using sasToken in place of storage key
.credentials(storageAccountName, sasToken)
.buildView(BlobStoreContext.class);
{code}
Since we use SAS token based authentication, so I used sasToken in place of
storageKey in the above code. But this throws :
*com.google.common.io.BaseEncoding$DecodingException: Invalid input length 121*
Please find the full stack trace attached for your reference.
I could be wrong, but after following the test class
[SharedKeyLiteAuthenticationTest.java|https://gitbox.apache.org/repos/asf?p=jclouds.git;a=blob;f=providers/azureblob/src/test/java/org/jclouds/azure/storage/filters/SharedKeyLiteAuthenticationTest.java;h=5e0c2264127a2665c70391f010c6031fabe27854;hb=0ce9261]
which was committed as a part of the
[PR|https://gitbox.apache.org/repos/asf?p=jclouds.git;h=0ce9261], I tried the
below code of my own:
{code:java}
BlobStoreContext context = ContextBuilder.newBuilder("azureblob")
.endpoint("https://${identity}.blob.core.windows.net;)
.credentials(accountName, sasToken)
.modules(new ArrayList<>(3)) //just added an empty list
.buildInjector()
.getInstance(BlobStoreContext.class);
{code}
The above code also throws the same exception.
Is it right to use SAS token as credential in place of storage key ?
Is it possible to use SAS token to configure BlobStoreContext for azure in
order to get the portable BlobStore ?
A code snippet would be highly appreciated.
Thanks and Regards,
Biswa Ranjan Ray
was (Author: roy.biswa):
Hi,
I'm trying to configure BlobStoreContext for Azure storage using SAS token. I
found the below code snippet from [jclouds
|https://jclouds.apache.org/guides/azure-storage/]site that offers the portable
BlobStore api:
{code:java}
BlobStoreContext context = ContextBuilder.newBuilder("azureblob")
.credentials(storageAccountName, sasToken) //using sasToken in place of
storage key
.buildView(BlobStoreContext.class);
{code}
Since we use SAS token based authentication, so I used sasToken in place of
storageKey in the above code. But this throws :
*com.google.common.io.BaseEncoding$DecodingException: Invalid input length 121*
Please find the full stack trace attached for your reference.
I could be wrong, but after following the test class
[SharedKeyLiteAuthenticationTest.java|https://gitbox.apache.org/repos/asf?p=jclouds.git;a=blob;f=providers/azureblob/src/test/java/org/jclouds/azure/storage/filters/SharedKeyLiteAuthenticationTest.java;h=5e0c2264127a2665c70391f010c6031fabe27854;hb=0ce9261]
which was committed as a part of the
[PR|https://gitbox.apache.org/repos/asf?p=jclouds.git;h=0ce9261], I tried the
below code of my own:
{code:java}
BlobStoreContext context = ContextBuilder.newBuilder("azureblob")
.endpoint("https://${identity}.blob.core.windows.net;)
.credentials(accountName, sasToken)
.modules(new ArrayList<>(3)) //just added an empty list of
size 3
.buildInjector()
.getInstance(BlobStoreContext.class);
{code}
The above code also throws the same exception.
Is it right to use SAS token as credential in place of storage key ?
Is it possible to use SAS token to configure BlobStoreContext for azure in
order to get the portable BlobStore ?
A code snippet would be highly appreciated.
Thanks and Regards,
Biswa Ranjan Ray
> Support for SAS token based Authentication for Azure Blob Storage
> -
>
> Key: JCLOUDS-1428
> URL: https://issues.apache.org/jira/browse/JCLOUDS-1428
> Project: jclouds
> Issue Type: Improvement
> Components: jclouds-blobstore
>Reporter: Himanshu Jain
>Priority: Major
> Labels: azureblob
> Fix For: 2.2.0, 2.1.3
>
>
> Hi,
> We have one use case where we want to provide limited access to objects in
> our storage accounts. We figured that the best way to do this is by using
> SAS token based authentication mechanism to upload/download objects to Azure
> Blob Storage - [SAS based
> Authentication|https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1]
> We found that JClouds client library provides support for Azure
[jira] [Comment Edited] (JCLOUDS-1428) Support for SAS token based Authentication for Azure Blob Storage
[ https://issues.apache.org/jira/browse/JCLOUDS-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737998#comment-16737998 ] Alexandra Horuszko edited comment on JCLOUDS-1428 at 1/10/19 10:01 AM: --- Hello Andrew, I am currently working on this task. I wonder, if I can somehow omit the URL encoding of the SAS URL in HttpRequest which is built in BlobRequestSigner interface? I would also like to ask for a slack channel invitation! Thanks > Support for SAS token based Authentication for Azure Blob Storage > - > > Key: JCLOUDS-1428 > URL: https://issues.apache.org/jira/browse/JCLOUDS-1428 > Project: jclouds > Issue Type: Improvement > Components: jclouds-blobstore >Reporter: Himanshu Jain >Priority: Major > Labels: azureblob > > Hi, > We have one use case where we want to provide limited access to objects in > our storage accounts. We figured that the best way to do this is by using > SAS token based authentication mechanism to upload/download objects to Azure > Blob Storage - [SAS based > Authentication|https://docs.microsoft.com/en-us/azure/storage/common/storage-dotnet-shared-access-signature-part-1] > We found that JClouds client library provides support for Azure Blob Storage > using account keys which might not fit our use case because of security > reasons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
