[jira] [Comment Edited] (YARN-10310) YARN Service - User is able to launch a service with same name
[
https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138637#comment-17138637
]
Bilwa S T edited comment on YARN-10310 at 6/17/20, 4:50 PM:
Hi [~eyang]
I could see that ServiceClient#submitapp adds YarnServiceConstants.APP_TYPE to
all applications .
{code:java}
submissionContext.setQueue(queue);
submissionContext.setApplicationName(serviceName);
submissionContext.setApplicationType(YarnServiceConstants.APP_TYPE);
{code}
I debugged it and found that yarnClient.getApplications(request) fails at below
case
{code:java}
if (users != null && !users.isEmpty() &&
!users.contains(application.getUser())) {
continue;
}
{code}
as applicationType was same but whereas users list had hdfs/had...@hadoop.com
and application.getuser() was hdfs.
Yes I think we should change appType also based on parameter. currently we are
adding yarn-service by default to all apps. Thanks
was (Author: bilwast):
Hi [~eyang]
I could see that ServiceClient#submitapp adds YarnServiceConstants.APP_TYPE to
all applications .
{code:java}
submissionContext.setQueue(queue);
submissionContext.setApplicationName(serviceName);
submissionContext.setApplicationType(YarnServiceConstants.APP_TYPE);
{code}
I debugged it and found that yarnClient.getApplications(request) fails at below
case
{code:java}
if (applicationTypes != null && !applicationTypes.isEmpty()) {
String appTypeToMatch =
StringUtils.toLowerCase(application.getApplicationType());
if (!applicationTypes.contains(appTypeToMatch)) {
continue;
}
}
if (applicationStates != null && !applicationStates.isEmpty()) {
if (!applicationStates.contains(application
.createApplicationState())) {
continue;
}
}
if (users != null && !users.isEmpty() &&
!*users.contains(application.getUser())*) {
continue;
}
{code}
as applicationType was same but whereas users list had hdfs/had...@hadoop.com
and application.getuser() was hdfs.
Yes I think we should change appType also based on parameter. currently we are
adding yarn-service by default to all apps. Thanks
> YARN Service - User is able to launch a service with same name
> --
>
> Key: YARN-10310
> URL: https://issues.apache.org/jira/browse/YARN-10310
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-10310.001.patch
>
>
> As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to
> get user whereas ClientRMService#submitApplication uses
> UserGroupInformation.getCurrentUser().getShortUserName() to set application
> username.
> In case of user with name hdfs/had...@hadoop.com. below condition fails
> ClientRMService#getApplications()
> {code:java}
> if (users != null && !users.isEmpty() &&
> !users.contains(application.getUser())) {
> continue;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10310) YARN Service - User is able to launch a service with same name
[
https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138573#comment-17138573
]
Eric Yang edited comment on YARN-10310 at 6/17/20, 3:56 PM:
[~BilwaST] The root cause is the parameter -appTypes unit-test.
Using hdfs/had...@example.com principal, the error message is same as using
h...@example.com.
{code}
[hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ ./bin/yarn app -launch abc sleeper
2020-06-17 08:17:17,867 INFO client.DefaultNoHARMFailoverProxyProvider:
Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032
2020-06-17 08:17:18,320 INFO client.DefaultNoHARMFailoverProxyProvider:
Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032
2020-06-17 08:17:18,323 INFO client.ApiServiceClient: Loading service
definition from local FS:
/usr/local/hadoop-3.4.0-SNAPSHOT/share/hadoop/yarn/yarn-service-examples/sleeper/sleeper.json
2020-06-17 08:17:21,104 INFO client.ApiServiceClient: Application ID:
application_1592406514799_0003
[hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ ./bin/yarn app -launch abc sleeper
2020-06-17 08:17:32,401 INFO client.DefaultNoHARMFailoverProxyProvider:
Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032
2020-06-17 08:17:32,971 INFO client.DefaultNoHARMFailoverProxyProvider:
Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032
2020-06-17 08:17:32,974 INFO client.ApiServiceClient: Loading service
definition from local FS:
/usr/local/hadoop-3.4.0-SNAPSHOT/share/hadoop/yarn/yarn-service-examples/sleeper/sleeper.json
2020-06-17 08:17:35,320 ERROR client.ApiServiceClient: Service name abc is
already taken.
{code}
verifyNoLiveAppInRM only look for appTypes == YarnServiceConstants.APP_TYPE.
The correct fix might be adding appTypes == unit-test to the
GetApplicationRequest to obtain the correct type of applications.
HDFS error message "Dir existing on hdfs." is to safe guard that a instance of
the yarn-service application in suspended mode (where there is no copy running
in RM), and its working directory exists. The error message is not wrong for
the suspended use case, and I agree that there might be better way to support
--appTypes flag for YARN service API to yield consistent output. Could you
refine the patch accordingly? Thanks
was (Author: eyang):
[~BilwaST] The root cause is the parameter -appTypes unit-test.
Using hdfs/had...@example.com principal, the error message is same as using
h...@example.com.
{code}
[hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ ./bin/yarn app -launch abc sleeper
2020-06-17 08:17:17,867 INFO client.DefaultNoHARMFailoverProxyProvider:
Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032
2020-06-17 08:17:18,320 INFO client.DefaultNoHARMFailoverProxyProvider:
Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032
2020-06-17 08:17:18,323 INFO client.ApiServiceClient: Loading service
definition from local FS:
/usr/local/hadoop-3.4.0-SNAPSHOT/share/hadoop/yarn/yarn-service-examples/sleeper/sleeper.json
2020-06-17 08:17:21,104 INFO client.ApiServiceClient: Application ID:
application_1592406514799_0003
[hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ ./bin/yarn app -launch abc sleeper
2020-06-17 08:17:32,401 INFO client.DefaultNoHARMFailoverProxyProvider:
Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032
2020-06-17 08:17:32,971 INFO client.DefaultNoHARMFailoverProxyProvider:
Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032
2020-06-17 08:17:32,974 INFO client.ApiServiceClient: Loading service
definition from local FS:
/usr/local/hadoop-3.4.0-SNAPSHOT/share/hadoop/yarn/yarn-service-examples/sleeper/sleeper.json
2020-06-17 08:17:35,320 ERROR client.ApiServiceClient: Service name abc is
already taken.
{code}
verifyNoLiveAppInRM only look for appTypes == YarnServiceConstants.APP_TYPE.
The correct fix might be adding appTypes == unit-test to the
GetApplicationRequest to obtain the correct type of applications.
HDFS error message "Dir existing on hdfs." is to safe guard that a instance of
the yarn-service application in suspended mode (where there is no copy running
in RM), and it's working directory. The error message is not wrong for the
suspended use case, and I agree that there might be better way to support
--appTypes flag for YARN service API to yield consistent output. Could you
refine the patch according it? Thanks
> YARN Service - User is able to launch a service with same name
> --
>
> Key: YARN-10310
> URL: https://issues.apache.org/jira/browse/YARN-10310
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-10310.001.patch
>
>
[jira] [Comment Edited] (YARN-10310) YARN Service - User is able to launch a service with same name
[
https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17137965#comment-17137965
]
Bilwa S T edited comment on YARN-10310 at 6/16/20, 10:47 PM:
-
[~eyang] Thanks for explaination. Let me explain my testing steps. I have not
deleted file from hdfs in below steps
1. Launch an application with user *hdfs* using below command two times with
same app name
{code:java}
./yarn app -launch sleeper-service
../share/hadoop/yarn/yarn-service-examples/sleeper/sleeper.json -appTypes
unit-test
{code}
App launch fails with error saying "*Failed to create service sleeper-service,
because it already exists.*"
2. Whereas when i launch an app with *hdfs/had...@hadoop.com* two times with
same app name
App launch fails due to *DIr existing on hdfs.*
This behaviour is not same. Please do check once. Thanks
was (Author: bilwast):
[~eyang] Thanks for explaination. Let me explain my testing steps. I have not
deleted file from hdfs in below steps
1. Launch an application with user hdfs using below command two times with same
app name
{code:java}
./yarn app -launch sleeper-service
../share/hadoop/yarn/yarn-service-examples/sleeper/sleeper.json -appTypes
unit-test
{code}
App launch fails with error saying "Failed to create service sleeper-service,
because it already exists."
2. Whereas when i launch an app with hdfs/had...@hadoop.com two times with same
app name
App launch fails due to DIr existing on hdfs.
This behaviour is not same. Please do check once. Thanks
> YARN Service - User is able to launch a service with same name
> --
>
> Key: YARN-10310
> URL: https://issues.apache.org/jira/browse/YARN-10310
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-10310.001.patch
>
>
> As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to
> get user whereas ClientRMService#submitApplication uses
> UserGroupInformation.getCurrentUser().getShortUserName() to set application
> username.
> In case of user with name hdfs/had...@hadoop.com. below condition fails
> ClientRMService#getApplications()
> {code:java}
> if (users != null && !users.isEmpty() &&
> !users.contains(application.getUser())) {
> continue;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10310) YARN Service - User is able to launch a service with same name
[
https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136352#comment-17136352
]
Bilwa S T edited comment on YARN-10310 at 6/16/20, 6:37 AM:
[~eyang] Actually the verifyNoLiveAppInRM check happens before it checks
service json from hdfs. I think you can compare the behaviour by submitting
with user hdfs and [hdfs/had...@example.com.|mailto:hdfs/had...@example.com.]
You can see below code
{code:java}
public ApplicationId actionCreate(Service service)
throws IOException, YarnException {
String serviceName = service.getName();
ServiceApiUtil.validateAndResolveService(service, fs, getConfig());
verifyNoLiveAppInRM(serviceName, "create");
Path appDir = checkAppNotExistOnHdfs(service, false);
// Write the definition first and then submit - AM will read the definition
ServiceApiUtil.createDirAndPersistApp(fs, appDir, service);
ApplicationId appId = submitApp(service);
cachedAppInfo.put(serviceName, new AppInfo(appId, service
.getKerberosPrincipal().getPrincipalName()));
service.setId(appId.toString());
// update app definition with appId
ServiceApiUtil.writeAppDefinition(fs, appDir, service);
return appId;
}
{code}
verifyNoLiveAppInRM is called before checkAppNotExistOnHdfs . so
verifyNoLiveAppInRM should have failed in your case . but it didnt
was (Author: bilwast):
[~eyang] Actually the verifyNoLiveAppInRM check happens before it checks
service json from hdfs. I think you can compare the behaviour by submitting
with user hdfs and hdfs/had...@example.com.
> YARN Service - User is able to launch a service with same name
> --
>
> Key: YARN-10310
> URL: https://issues.apache.org/jira/browse/YARN-10310
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-10310.001.patch
>
>
> As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to
> get user whereas ClientRMService#submitApplication uses
> UserGroupInformation.getCurrentUser().getShortUserName() to set application
> username.
> In case of user with name hdfs/had...@hadoop.com. below condition fails
> ClientRMService#getApplications()
> {code:java}
> if (users != null && !users.isEmpty() &&
> !users.contains(application.getUser())) {
> continue;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10310) YARN Service - User is able to launch a service with same name
[
https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17130066#comment-17130066
]
Bilwa S T edited comment on YARN-10310 at 6/10/20, 4:51 AM:
Hi [~eyang]
Testing done for this with username hdfs/had...@hadoop.com. Getting below
exception
{code:java}
Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException:
Failed to create service sleeper-service, because it already exists.
at
org.apache.hadoop.yarn.service.client.ServiceClient.verifyNoLiveAppInRM(ServiceClient.java:947)
at
org.apache.hadoop.yarn.service.client.ServiceClient.actionCreate(ServiceClient.java:540)
at
org.apache.hadoop.yarn.service.client.ServiceClient.actionLaunch(ServiceClient.java:531)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:574)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:125)
{code}
was (Author: bilwast):
Testing done for this. Getting below exception
{code:java}
Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException:
Failed to create service sleeper-service, because it already exists.
at
org.apache.hadoop.yarn.service.client.ServiceClient.verifyNoLiveAppInRM(ServiceClient.java:947)
at
org.apache.hadoop.yarn.service.client.ServiceClient.actionCreate(ServiceClient.java:540)
at
org.apache.hadoop.yarn.service.client.ServiceClient.actionLaunch(ServiceClient.java:531)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:574)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:125)
{code}
> YARN Service - User is able to launch a service with same name
> --
>
> Key: YARN-10310
> URL: https://issues.apache.org/jira/browse/YARN-10310
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-10310.001.patch
>
>
> As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to
> get user whereas ClientRMService#submitApplication uses
> UserGroupInformation.getCurrentUser().getShortUserName() to set application
> username.
> In case of user with name hdfs/had...@hadoop.com. below condition fails
> ClientRMService#getApplications()
> {code:java}
> if (users != null && !users.isEmpty() &&
> !users.contains(application.getUser())) {
> continue;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10310) YARN Service - User is able to launch a service with same name
[
https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17130066#comment-17130066
]
Bilwa S T edited comment on YARN-10310 at 6/10/20, 4:50 AM:
Testing done for this. Getting below exception
{code:java}
Exception in thread "main" org.apache.hadoop.yarn.exceptions.YarnException:
Failed to create service sleeper-service, because it already exists.
at
org.apache.hadoop.yarn.service.client.ServiceClient.verifyNoLiveAppInRM(ServiceClient.java:947)
at
org.apache.hadoop.yarn.service.client.ServiceClient.actionCreate(ServiceClient.java:540)
at
org.apache.hadoop.yarn.service.client.ServiceClient.actionLaunch(ServiceClient.java:531)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.run(ApplicationCLI.java:574)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at
org.apache.hadoop.yarn.client.cli.ApplicationCLI.main(ApplicationCLI.java:125)
{code}
was (Author: bilwast):
Testing done for this.
> YARN Service - User is able to launch a service with same name
> --
>
> Key: YARN-10310
> URL: https://issues.apache.org/jira/browse/YARN-10310
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-10310.001.patch
>
>
> As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to
> get user whereas ClientRMService#submitApplication uses
> UserGroupInformation.getCurrentUser().getShortUserName() to set application
> username.
> In case of user with name hdfs/had...@hadoop.com. below condition fails
> ClientRMService#getApplications()
> {code:java}
> if (users != null && !users.isEmpty() &&
> !users.contains(application.getUser())) {
> continue;
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
