[
https://issues.apache.org/jira/browse/HADOOP-17853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17405791#comment-17405791
]
Sneha Vijayarajan commented on HADOOP-17853:
--------------------------------------------
Hi [[email protected]],
Feature update in ABFS driver is to pass Http Request URL, queryparams and
headers to APIs exposed by a maven artifact that takes up the responsibility to
effectively use the feature path for read. In addition to the changes to
redirect the requests, a class which holds on to the session details will also
be added. The feature will be in dormant state and will have to be enabled
consciously in environments that are suitable for the feature. One such
requirements is that the Hadoop cluster needs to be on Azure VM.
The test update does not change or reduce the existing tests that run over
REST. The read related tests for the feature are triggered separately for the
respective test scenario. Considering devs on ABFS driver need not have access
to feature enabled environment, all the read tests validating the feature will
run even if the feature flag is off and will rely on test mock code to
replicate feature flow. This will ensure that future checkins by any dev will
not break REST or the feature path.
On an actual store connected environment, close to 500 looped test runs have
been completed successfully with no functional failures. This test run count is
post the feature code freeze in ABFS driver.
Counters that IOStatistics collects remain valid while the feature is
exercised.
If the feature were to be enabled on an environment that isnt suitable to
utilize it, request will fallback to REST and for the lifetime of that
InputStream instance reads will only be attempted on REST. Fallbacks are in
place even for the case should feature hit an irrecoverable issue. Hence
callers are not exposed to any new error scenarios than what already exists.
Store connectivity over REST will continue to be a workflow that we optimize
and support.
Asserts in all the new tests added will use AssertJ.
I have closed the PR that I had initially linked to this JIRA. The changes
included few test fixes and some refactoring needed for the feature as well.
[~sumangala] has split the test fixes that we had to 4 PRs. Code updates that
intent to refactor and make it easy to add the feature are split into 2
different PRs. JIRAs of those PRs have been added here as children. Once these
get checked in, we are left with just 1 PR that brings in the feature code.
The pending change is as atomic as possible for the feature. Committing it to a
separate branch will need us to commit to trunk subsequently with no more add
ons from the driver. Hence I would prefer to raise the PR on trunk directly.
> ABFS: Enable optional store connectivity over azure specific protocol for
> data egress
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-17853
> URL: https://issues.apache.org/jira/browse/HADOOP-17853
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.4.0
> Reporter: Sneha Vijayarajan
> Assignee: Sneha Vijayarajan
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This Jira is to provide an option to enable store access on read path over an
> Azure specific protocol. This will only work on Azure VMs and hence will be
> disabled by default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
