[
https://issues.apache.org/jira/browse/HADOOP-19178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17852143#comment-17852143
]
ASF GitHub Bot commented on HADOOP-19178:
-----------------------------------------
hadoop-yetus commented on PR #6862:
URL: https://github.com/apache/hadoop/pull/6862#issuecomment-2148102249
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 88m 10s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 26s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 133m 50s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6862/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6862 |
| Optional Tests | dupname asflicense mvnsite codespell detsecrets
markdownlint |
| uname | Linux 5c697d97c1bc 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3bf2283c6aa00544ba9ed574f7bf52322fb74be3 |
| Max. process+thread count | 549 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6862/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> WASB Driver Deprecation and eventual removal
> --------------------------------------------
>
> Key: HADOOP-19178
> URL: https://issues.apache.org/jira/browse/HADOOP-19178
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.4.0
> Reporter: Sneha Vijayarajan
> Assignee: Sneha Vijayarajan
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.1
>
>
> *WASB Driver*
> WASB driver was developed to support FNS (FlatNameSpace) Azure Storage
> accounts. FNS accounts do not honor File-Folder syntax. HDFS Folder
> operations hence are mimicked at client side by WASB driver and certain
> folder operations like Rename and Delete can lead to lot of IOPs with
> client-side enumeration and orchestration of rename/delete operation blob by
> blob. It was not ideal for other APIs too as initial checks for path is a
> file or folder needs to be done over multiple metadata calls. These led to a
> degraded performance.
> To provide better service to Analytics customers, Microsoft released ADLS
> Gen2 which are HNS (Hierarchical Namespace) , i.e File-Folder aware store.
> ABFS driver was designed to overcome the inherent deficiencies of WASB and
> customers were informed to migrate to ABFS driver.
> *Customers who still use the legacy WASB driver and the challenges they face*
> Some of our customers have not migrated to the ABFS driver yet and continue
> to use the legacy WASB driver with FNS accounts.
> These customers face the following challenges:
> * They cannot leverage the optimizations and benefits of the ABFS driver.
> * They need to deal with the compatibility issues should the files and
> folders were modified with the legacy WASB driver and the ABFS driver
> concurrently in a phased transition situation.
> * There are differences for supported features for FNS and HNS over ABFS
> Driver
> * In certain cases, they must perform a significant amount of re-work on
> their workloads to migrate to the ABFS driver, which is available only on HNS
> enabled accounts in a fully tested and supported scenario.
> *Deprecation plans for WASB*
> We are introducing a new feature that will enable the ABFS driver to support
> FNS accounts (over BlobEndpoint) using the ABFS scheme. This feature will
> enable customers to use the ABFS driver to interact with data stored in GPv2
> (General Purpose v2) storage accounts.
> With this feature, the customers who still use the legacy WASB driver will be
> able to migrate to the ABFS driver without much re-work on their workloads.
> They will however need to change the URIs from the WASB scheme to the ABFS
> scheme.
> Once ABFS driver has built FNS support capability to migrate WASB customers,
> WASB driver will be declared deprecated in OSS documentation and marked for
> removal in next major release. This will remove any ambiguity for new
> customer onboards as there will be only one Microsoft driver for Azure
> Storage and migrating customers will get SLA bound support for driver and
> service, which was not guaranteed over WASB.
> We anticipate that this feature will serve as a stepping stone for customers
> to move to HNS enabled accounts with the ABFS driver, which is our
> recommended stack for big data analytics on ADLS Gen2.
> *Any Impact for* *existing customers who are using ADLS Gen2 (HNS enabled
> account) with ABFS driver* *?*
> This feature does not impact the existing customers who are using ADLS Gen2
> (HNS enabled account) with ABFS driver.
> They do not need to make any changes to their workloads or configurations.
> They will still enjoy the benefits of HNS, such as atomic operations,
> fine-grained access control, scalability, and performance.
> *Official recommendation*
> Microsoft continues to recommend all Big Data and Analytics customers to use
> Azure Data Lake Gen2 (ADLS Gen2) using the ABFS driver and will continue to
> optimize this scenario in future, we believe that this new option will help
> all those customers to transition to a supported scenario immediately, while
> they plan to ultimately move to ADLS Gen2 (HNS enabled account).
> *New Authentication options that a WASB to ABFS Driver migrating customer
> will get*
> Below auth types that WASB provides will continue to work on the new FNS over
> ABFS Driver over configuration that accepts these SAS types (similar to WASB)
> * SharedKey
> * Account SAS
> * Service/Container SAS
> Below authentication types that were not supported by WASB driver but
> supported by ABFS driver will continue to be available for new FNS over ABFS
> Driver
> * OAuth 2.0 Client Credentials
> * OAuth 2.0: Refresh Token
> * Azure Managed Identity
> * Custom OAuth 2.0 Token Provider
> ABFS Driver SAS Token Provider plugin present today for UserDelegation SAS
> and Directly SAS will continue to work only for HNS accounts.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
