[
https://issues.apache.org/jira/browse/YARN-9920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16977243#comment-16977243
]
Prabhu Joseph commented on YARN-9920:
-------------------------------------
*Changes done as part of Patch-04*
# Added remoteAddress parameter in Queue#hasAccess and Scheduler#checkAccess.
This is passed to QueueACLsManager
which sends it to Ranger Auth Provider as part of AccessRequest.
# Added RMWebAppFilter#getRemoteAddress which provides the http client address
for every http thread similar to IPC Server.
# Added a wrapper method ClientRMService#getRemoteAddress which provides the
client address. This gets from
RMWebAppFilter#getRemoteAddress in case of http thread or
Server#getRemoteAddress in case of IPC Server handler thread.
# RMApp stores the client address of job submitter which is used to check
access later as part of EventDispatcher thread.
*Testing done - remoteAddress in checkAccess is verified as part of below
operations.*
1. IPC Client testing:
{code:java}
Submit Application:
yarn jar
/HADOOP/hadoop-3.3.0-SNAPSHOT/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.0-SNAPSHOT.jar
pi 1 1
Get Apps:
yarn application -list
Move Queue:
yarn application -changeQueue dummy -appId application_1574149031529_0001
Update App Lifetime:
yarn application -updateLifetime 100000 -appId application_1574149031529_0001
{code}
2. RMWebService testing:
{code:java}
Move Queue:
curl -H 'Content-Type: application/json' -H 'Accept: application/json' -X PUT
--data '{"queue": "default"}' --negotiate -u:
'http://yarndocker-3:8088/ws/v1/cluster/apps/application_1574149031529_0001/queue'
Kill Application:
curl -H 'Content-Type: application/json' -H 'Accept: application/json' -X PUT
--data '{"state": "KILLED"}' --negotiate -u:
'http://yarndocker-3:8088/ws/v1/cluster/apps/application_1574099628047_0002/state
Get Apps:
curl --negotiate -u : 'http://yarndocker-3:8088/ws/v1/cluster/apps'
YARN UI:
http://yarndocker-3:8088/cluster/scheduler
{code}
3. Recovery with older apps (ones without remoteAddress) and new ones (with
remoteAddress).
> YarnAuthorizationProvider AccessRequest gets Null RemoteAddress from
> FairScheduler
> ----------------------------------------------------------------------------------
>
> Key: YARN-9920
> URL: https://issues.apache.org/jira/browse/YARN-9920
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, security
> Affects Versions: 3.3.0
> Reporter: Prabhu Joseph
> Assignee: Prabhu Joseph
> Priority: Major
> Attachments: AccessAudist_yarn_clientIPempty.png,
> YARN-9920-001.patch, YARN-9920-002.patch, YARN-9920-003.patch,
> YARN-9920-004.patch
>
>
> YarnAuthorizationProvider AccessRequest has null RemoteAddress in case of
> FairScheduler. FSQueue#hasAccess uses Server.getRemoteAddress() which will be
> null when the call is from RMWebServices and EventDispatcher. It works fine
> when called by IPC Server Handler.
> FSQueue#hasAccess is called at three places where (2) and (3) returns null.
> *1. IPC Server -> RMAppManager#createAndPopulateNewRMApp -> FSQueue#hasAccess
> -> Server.getRemoteAddress returns correct Remote IP.*
>
> *2. IPC Server -> RMAppManager#createAndPopulateNewRMApp ->
> AppAddedSchedulerEvent*
> *EventDispatcher -> FairScheduler#addApplication -> FSQueue.hasAccess ->
> Server.getRemoteAddress returns null*
>
> {code:java}
> org.apache.hadoop.yarn.security.ConfiguredYarnAuthorizer.checkPermission(ConfiguredYarnAuthorizer.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue.hasAccess(FSQueue.java:316)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.addApplication(FairScheduler.java:509)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1268)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:133)
> at
> org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
> {code}
>
> *3. RMWebServices -> QueueACLsManager#checkAccess -> FSQueue.hasAccess ->
> Server.getRemoteAddress returns null.*
> {code:java}
> org.apache.hadoop.yarn.security.ConfiguredYarnAuthorizer.checkPermission(ConfiguredYarnAuthorizer.java:101)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSQueue.hasAccess(FSQueue.java:316)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.checkAccess(FairScheduler.java:1610)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.QueueACLsManager.checkAccess(QueueACLsManager.java:84)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.hasAccess(RMWebServices.java:270)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.getApps(RMWebServices.java:553)
> {code}
>
> Have verified with CapacityScheduler and it works fine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
