[ https://issues.apache.org/jira/browse/HADOOP-16005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16774405#comment-16774405 ]
Clemens Wolff commented on HADOOP-16005: ---------------------------------------- [~DanielZhou] Thanks for the comments. Some notes: 1) I've pushed some changes to the branch to address your comments. 2) Could you point me to the breaking commit that you mentioned? I rebased my local branch onto trunk (at 6c4ab03) and don't see any conflicts. 3) When running {{mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify}} on trunk, I get a [bunch of errors|https://user-images.githubusercontent.com/1086421/53193268-9a584a00-35de-11e9-90a0-95781ed3bda1.png]. For reference, here's my [azure-auth-keys.xml|https://user-images.githubusercontent.com/1086421/53193465-08047600-35df-11e9-974d-8b2d3fb202c8.png]. > NativeAzureFileSystem does not support setXAttr > ----------------------------------------------- > > Key: HADOOP-16005 > URL: https://issues.apache.org/jira/browse/HADOOP-16005 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Reporter: Clemens Wolff > Assignee: Clemens Wolff > Priority: Major > > When interacting with Azure Blob Storage via the Hadoop FileSystem client, > it's currently (as of > [a8bbd81|https://github.com/apache/hadoop/commit/a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b]) > not possible to set custom metadata attributes. > Here is a snippet that demonstrates the missing behavior (throws an > UnsupportedOperationException): > {code:java} > val blobAccount = "SET ME" > val blobKey = "SET ME" > val blobContainer = "SET ME" > val blobFile = "SET ME" > import org.apache.hadoop.conf.Configuration > import org.apache.hadoop.fs.{FileSystem, Path} > val conf = new Configuration() > conf.set("fs.wasbs.impl", "org.apache.hadoop.fs.azure.NativeAzureFileSystem") > conf.set(s"fs.azure.account.key.$blobAccount.blob.core.windows.net", blobKey) > val path = new > Path(s"wasbs://$blobContainer@$blobAccount.blob.core.windows.net/$blobFile") > val fs = FileSystem.get(path, conf) > fs.setXAttr(path, "somekey", "somevalue".getBytes) > {code} > Looking at the code in hadoop-tools/hadoop-azure, NativeAzureFileSystem > inherits the default setXAttr from FileSystem which throws the > UnsupportedOperationException. > The underlying Azure Blob Storage service does support custom metadata > ([service > docs|https://docs.microsoft.com/en-us/azure/storage/blobs/storage-properties-metadata]) > as does the azure-storage SDK that's being used by NativeAzureFileSystem > ([SDK > docs|http://javadox.com/com.microsoft.azure/azure-storage/2.0.0/com/microsoft/azure/storage/blob/CloudBlob.html#setMetadata(java.util.HashMap)]). > Is there another way that I should be setting custom metadata on Azure Blob > Storage files? Is there a specific reason why setXAttr hasn't been > implemented on NativeAzureFileSystem? If not, I can take a shot at > implementing it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org