[
https://issues.apache.org/jira/browse/HADOOP-19232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17911323#comment-17911323
]
ASF GitHub Bot commented on HADOOP-19232:
-----------------------------------------
anujmodi2021 commented on code in PR #7272:
URL: https://github.com/apache/hadoop/pull/7272#discussion_r1908149084
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsPaginatedDelete.java:
##########
@@ -142,6 +143,8 @@ private void setTestUserConf(Configuration conf, String
key, String value) {
*/
@Test
public void testRecursiveDeleteWithPagination() throws Exception {
+ Assume.assumeTrue(
Review Comment:
we can use base class method to check service type
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsOutputStream.java:
##########
@@ -253,4 +302,127 @@ private AbfsOutputStream getStream() throws
URISyntaxException, IOException {
return createAbfsOutputStreamWithFlushEnabled(fs1, pathFs1);
}
+ /**
+ * Verify that if getBlockList throws exception append should fail.
+ */
+ @Test
+ public void testValidateGetBlockList() throws Exception {
+ AzureBlobFileSystem fs = Mockito.spy(getFileSystem());
+ Assume.assumeTrue(!getIsNamespaceEnabled(fs));
+ AzureBlobFileSystemStore store = Mockito.spy(fs.getAbfsStore());
+ Assume.assumeTrue(store.getClient() instanceof AbfsBlobClient);
Review Comment:
`assumeBlobEndpoint()`
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/AzcopyExecutionException.java:
##########
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.io.IOException;
+
+public class AzcopyExecutionException extends IOException {
Review Comment:
Javadoc for this class, specifying only used in test code
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFilesystemAcl.java:
##########
@@ -1042,6 +1043,7 @@ public void testSetPermissionOnlyDefault() throws
Exception {
public void testDefaultAclNewFile() throws Exception {
final AzureBlobFileSystem fs = this.getFileSystem();
assumeTrue(getIsNamespaceEnabled(fs));
+ Assume.assumeTrue(fs.getAbfsStore().getClientHandler().getIngressClient()
instanceof AbfsDfsClient);
Review Comment:
Same here as well, we can assert on getAbfsServiceType()
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsOutputStream.java:
##########
@@ -253,4 +302,127 @@ private AbfsOutputStream getStream() throws
URISyntaxException, IOException {
return createAbfsOutputStreamWithFlushEnabled(fs1, pathFs1);
}
+ /**
+ * Verify that if getBlockList throws exception append should fail.
+ */
+ @Test
+ public void testValidateGetBlockList() throws Exception {
+ AzureBlobFileSystem fs = Mockito.spy(getFileSystem());
+ Assume.assumeTrue(!getIsNamespaceEnabled(fs));
+ AzureBlobFileSystemStore store = Mockito.spy(fs.getAbfsStore());
+ Assume.assumeTrue(store.getClient() instanceof AbfsBlobClient);
+
+ // Mock the clientHandler to return the blobClient when getBlobClient is
called
+ AbfsClientHandler clientHandler = Mockito.spy(store.getClientHandler());
+ AbfsBlobClient blobClient = Mockito.spy(clientHandler.getBlobClient());
+
+ Mockito.doReturn(clientHandler).when(store).getClientHandler();
+ Mockito.doReturn(blobClient).when(clientHandler).getBlobClient();
+ Mockito.doReturn(blobClient).when(clientHandler).getIngressClient();
+
+ Mockito.doReturn(store).when(fs).getAbfsStore();
+ Path testFilePath = new Path("/testFile");
+ AbfsOutputStream os = Mockito.spy((AbfsOutputStream)
fs.create(testFilePath).getWrappedStream());
+
+ Mockito.doReturn(clientHandler).when(os).getClientHandler();
+ Mockito.doReturn(blobClient).when(clientHandler).getBlobClient();
+
+ AbfsRestOperationException exception =
getMockAbfsRestOperationException(HTTP_CONFLICT);
+ // Throw exception when getBlockList is called
+
Mockito.doThrow(exception).when(blobClient).getBlockList(Mockito.anyString(),
Mockito.any(TracingContext.class));
+
+ // Create a non-empty file
+ os.write(10);
Review Comment:
`TEN`
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsHttpClientRequestExecutor.java:
##########
@@ -61,6 +65,12 @@ public ITestAbfsHttpClientRequestExecutor() throws Exception
{
public void testExpect100ContinueHandling() throws Exception {
AzureBlobFileSystem fs = getFileSystem();
Path path = new Path("/testExpect100ContinueHandling");
+ if (isAppendBlobEnabled()) {
+ Assume.assumeFalse("Not valid for AppendBlob with blob endpoint",
Review Comment:
We can check using getAbfsServiceType()
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/services/ITestAbfsOutputStream.java:
##########
@@ -174,28 +187,39 @@ public void testAbfsOutputStreamClosingFsBeforeStream()
fs.close();
// verify that output stream close after fs.close() would raise a
// pathIOE containing the path being written to.
- LambdaTestUtils
- .intercept(PathIOException.class, getMethodName(), out::close);
+ intercept(PathIOException.class, getMethodName(), out::close);
}
}
@Test
public void testExpect100ContinueFailureInAppend() throws Exception {
+ if (!getIsNamespaceEnabled(getFileSystem())) {
+ Assume.assumeFalse("Not valid for APPEND BLOB", isAppendBlobEnabled());
+ }
Configuration configuration = new Configuration(getRawConfiguration());
configuration.set(FS_AZURE_ACCOUNT_IS_EXPECT_HEADER_ENABLED, "true");
AzureBlobFileSystem fs = getFileSystem(configuration);
Path path = new Path("/testFile");
AbfsOutputStream os = Mockito.spy(
(AbfsOutputStream) fs.create(path).getWrappedStream());
- AbfsClient spiedClient = Mockito.spy(os.getClient());
+ AzureIngressHandler ingressHandler = Mockito.spy(
+ os.getIngressHandler());
+ Mockito.doReturn(ingressHandler).when(os).getIngressHandler();
+
+ AbfsClient spiedClient = Mockito.spy(ingressHandler.getClient());
+ Mockito.doReturn(spiedClient).when(ingressHandler).getClient();
AbfsHttpOperation[] httpOpForAppendTest = new AbfsHttpOperation[2];
mockSetupForAppend(httpOpForAppendTest, spiedClient);
Mockito.doReturn(spiedClient).when(os).getClient();
fs.delete(path, true);
os.write(1);
- LambdaTestUtils.intercept(FileNotFoundException.class, () -> {
- os.close();
- });
+ if (spiedClient instanceof AbfsDfsClient) {
+ intercept(FileNotFoundException.class, os::close);
+ } else {
Review Comment:
What is the exact difference in behavior between DFS and Blob here?
Can we assert on status code as well??
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/AzcopyExecutionException.java:
##########
@@ -0,0 +1,33 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs.azurebfs.utils;
+
+import java.io.IOException;
+
+public class AzcopyExecutionException extends IOException {
+ private static final String SUGGESTION = "Try deleting the following azcopy
tool directory and rerun tests: ";
+
+ public AzcopyExecutionException(String message, String azcopyPath) {
+ super(message + SUGGESTION + azcopyPath);
+ }
+
+ public AzcopyExecutionException(String message, String azcopyPath, Throwable
cause) {
+ super(message + SUGGESTION + azcopyPath, cause);
+ }
+}
Review Comment:
EOF Error here and other classes.
> ABFS: [FnsOverBlob] Implementing Ingress Support with various Fallback
> Handling
> -------------------------------------------------------------------------------
>
> Key: HADOOP-19232
> URL: https://issues.apache.org/jira/browse/HADOOP-19232
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.4.0
> Reporter: Descifrado
> Assignee: Anmol Asrani
> Priority: Major
> Labels: pull-request-available
>
> Scope of this task is to refactor the AbfsOutputStream class to handle the
> ingress for DFS and Blob endpoint effectively.
> More details will be added soon.
> Perquisites for this Patch:
> 1. [HADOOP-19187] ABFS: [FnsOverBlob]Making AbfsClient Abstract for
> supporting both DFS and Blob Endpoint - ASF JIRA (apache.org)
> 2. [HADOOP-19226] ABFS: [FnsOverBlob]Implementing Azure Rest APIs on Blob
> Endpoint for AbfsBlobClient - ASF JIRA (apache.org)
> 3. [HADOOP-19207] ABFS: [FnsOverBlob]Response Handling of Blob Endpoint APIs
> and Metadata APIs - ASF JIRA (apache.org)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
