[
https://issues.apache.org/jira/browse/HADOOP-19778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18055134#comment-18055134
]
ASF GitHub Bot commented on HADOOP-19778:
-----------------------------------------
anujmodi2021 commented on code in PR #8207:
URL: https://github.com/apache/hadoop/pull/8207#discussion_r2741201388
##########
hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md:
##########
@@ -12,7 +12,15 @@
limitations under the License. See accompanying LICENSE file.
-->
-# Testing the Azure WASB client
+# Testing the Azure ABFS client
+
+Azure Data Lake Storage Gen 2 (ADLS Gen 2) is a set of capabilities dedicated
to
+big data analytics, built on top of Azure Blob Storage. The ABFS and ABFSS
+schemes target the ADLS Gen 2 REST API now having support for both HNS and FNS
Accounts.
+ADLS Gen 2 with HNS Enabled using DFS Endpoint offers better performance and
+scalability. ADLS Gen 2 also offers authentication and authorization
compatible
Review Comment:
Taken
##########
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/utils/AbfsTestUtils.java:
##########
@@ -132,4 +97,119 @@ public static void createFiles(AzureBlobFileSystem fs,
Path path, int numFiles)
}
executorService.shutdown();
}
-}
+
+ /**
+ * Assume that a condition is met. If not: log at WARN and
+ * then throw an {@link TestAbortedException}.
+ * @param message message in an assumption
+ * @param condition condition to probe
+ */
+ public static void assume(String message, boolean condition) {
+ if (!condition) {
+ LOG.warn(message);
+ }
+ assumeThat(condition).as(message).isTrue();
+ }
+
+ /**
+ * Get a string test property.
+ * <ol>
+ * <li>Look up configuration value (which can pick up core-default.xml),
+ * using {@code defVal} as the default value (if conf != null).
+ * </li>
+ * <li>Fetch the system property.</li>
+ * <li>If the system property is not empty or "(unset)":
+ * it overrides the conf value.
+ * </li>
+ * </ol>
+ * This puts the build properties in charge of everything. It's not a
+ * perfect design; having maven set properties based on a file, as ant let
+ * you do, is better for customization.
+ *
+ * As to why there's a special (unset) value, see
+ * @see <a
href="http://stackoverflow.com/questions/7773134/null-versus-empty-arguments-in-maven";>
+ * Stack Overflow</a>
+ * @param conf config: may be null
+ * @param key key to look up
+ * @param defVal default value
+ * @return the evaluated test property.
+ */
+
+ public static String getTestProperty(Configuration conf,
+ String key,
+ String defVal) {
+ String confVal = conf != null
+ ? conf.getTrimmed(key, defVal)
+ : defVal;
+ String propval = System.getProperty(key);
+ return StringUtils.isNotEmpty(propval) && !UNSET_PROPERTY.equals(propval)
+ ? propval : confVal;
+ }
+
+ /**
+ * Get a long test property.
+ * <ol>
+ * <li>Look up configuration value (which can pick up core-default.xml),
+ * using {@code defVal} as the default value (if conf != null).
+ * </li>
+ * <li>Fetch the system property.</li>
+ * <li>If the system property is not empty or "(unset)":
+ * it overrides the conf value.
+ * </li>
+ * </ol>
+ * This puts the build properties in charge of everything. It's not a
+ * perfect design; having maven set properties based on a file, as ant let
+ * you do, is better for customization.
+ *
+ * As to why there's a special (unset) value, see
+ * {@link <a
href="http://stackoverflow.com/questions/7773134/null-versus-empty-arguments-in-maven";>...</a>}
+ * @param conf config: may be null
+ * @param key key to look up
+ * @param defVal default value
+ * @return the evaluated test property.
+ */
+ public static long getTestPropertyLong(Configuration conf,
+ String key, long defVal) {
+ return Long.valueOf(
+ getTestProperty(conf, key, Long.toString(defVal)));
+ }
+
+ /**
+ * Get an integer test property; algorithm described in
+ * {@link #getTestPropertyLong(Configuration, String, long)}.
+ * @param key key to look up
+ * @param defVal default value
+ * @return the evaluated test property.
+ */
+ public static int getTestPropertyInt(Configuration conf,
+ String key, int defVal) {
+ return (int) getTestPropertyLong(conf, key, defVal);
+ }
+
+ /**
+ * Get a boolean test property; algorithm described in
+ * {@link #getTestPropertyLong(Configuration, String, long)}.
+ * @param key key to look up
+ * @param defVal default value
+ * @return the evaluated test property.
+ */
+ public static boolean getTestPropertyBool(Configuration conf,
+ String key,
+ boolean defVal) {
+ return Boolean.valueOf(
+ getTestProperty(conf, key, Boolean.toString(defVal)));
+ }
+
+ /**
+ * Assume that the scale tests are enabled by the relevant system property.
+ */
+ public static void assumeScaleTestsEnabled(Configuration conf) {
+ boolean enabled = getTestPropertyBool(
+ conf,
+ FS_AZURE_SCALE_TEST_ENABLED,
+ DEFAULT_SCALE_TESTS_ENABLED);
+ assume("Scale test disabled: to enable set property "
+ + FS_AZURE_SCALE_TEST_ENABLED,
+ enabled);
+ }
+}
Review Comment:
Taken
##########
hadoop-tools/hadoop-azure/src/site/markdown/deprecated_wasb.md:
##########
@@ -14,551 +14,14 @@
# Azure Blob Storage Support by Deprecated WASB Driver
+### WASB Driver has been deprecated and removed from official hadoop releases
starting from hadoop-3.5.0.
Review Comment:
Taken
> Remove Deprecated WASB Code from Hadoop
> ---------------------------------------
>
> Key: HADOOP-19778
> URL: https://issues.apache.org/jira/browse/HADOOP-19778
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.4.2
> Reporter: Anuj Modi
> Assignee: Anuj Modi
> Priority: Major
> Labels: pull-request-available
>
> WASB Driver has been deprecated for a while now:
> https://issues.apache.org/jira/browse/HADOOP-19178
> Starting with hadoop-3.5.0 we will no longer ship WASB Driver as a official
> hadoop-connector for Azure.
> Moving ahead ABFS Driver will be the only official Hadoop connector for both
> types of Accounts that is "Heirarchichal Namespace Enabled (HNS)" and
> "Heirarchichal Namespace Disabled (FNS)"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
