[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148834#comment-17148834 ] Bilwa S T commented on YARN-10310: -- [~eyang] This issue can be closed then. Thanks > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148063#comment-17148063 ] Eric Yang commented on YARN-10310: -- ServiceClient is designed to work with YARN service API. The main distinction between YARN service app and YARN app are: 1. YARN app can start multiple instances of the same name. 2. YARN service app can not start multiple instances of the same name. Application name and user name are used to construct DNS hostnames. They must be uniquely defined per service. 3. Using appType=unit-test with YARN service can result in conflict behavior between YARN app and YARN service. 4. YARN service app can be completely suspended with footprint in HDFS only. Classic YARN app does not handle suspension and resume. Those are the main differences between classic YARN app, and YARN services. Classic YARN API may not fully support YARN service, and the same vice versa due to the intersection of commonality of features is small between two mutually exclusive ideas. Neither classic YARN API nor YARN service API can operate as a union API calls of either type of applications. Hence, support appType=unit-test in YARN service client will create ambiguity that the classic YARN code was not designed to handle. I am inclined to close this as won't fix to prevent using appType=unit-test with YARN service api. > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17138637#comment-17138637 ] Bilwa S T commented on YARN-10310: -- Hi [~eyang] I could see that ServiceClient#submitapp adds YarnServiceConstants.APP_TYPE to all applications . {code:java} submissionContext.setQueue(queue); submissionContext.setApplicationName(serviceName); submissionContext.setApplicationType(YarnServiceConstants.APP_TYPE); {code} I debugged it and found that yarnClient.getApplications(request) fails at below case {code:java} if (applicationTypes != null && !applicationTypes.isEmpty()) { String appTypeToMatch = StringUtils.toLowerCase(application.getApplicationType()); if (!applicationTypes.contains(appTypeToMatch)) { continue; } } if (applicationStates != null && !applicationStates.isEmpty()) { if (!applicationStates.contains(application .createApplicationState())) { continue; } } if (users != null && !users.isEmpty() && !*users.contains(application.getUser())*) { continue; } {code} as applicationType was same but whereas users list had hdfs/had...@hadoop.com and application.getuser() was hdfs. Yes I think we should change appType also based on parameter. currently we are adding yarn-service by default to all apps. Thanks > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17138573#comment-17138573 ] Eric Yang commented on YARN-10310: -- [~BilwaST] The root cause is the parameter -appTypes unit-test. Using hdfs/had...@example.com principal, the error message is same as using h...@example.com. {code} [hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ ./bin/yarn app -launch abc sleeper 2020-06-17 08:17:17,867 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032 2020-06-17 08:17:18,320 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032 2020-06-17 08:17:18,323 INFO client.ApiServiceClient: Loading service definition from local FS: /usr/local/hadoop-3.4.0-SNAPSHOT/share/hadoop/yarn/yarn-service-examples/sleeper/sleeper.json 2020-06-17 08:17:21,104 INFO client.ApiServiceClient: Application ID: application_1592406514799_0003 [hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ ./bin/yarn app -launch abc sleeper 2020-06-17 08:17:32,401 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032 2020-06-17 08:17:32,971 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032 2020-06-17 08:17:32,974 INFO client.ApiServiceClient: Loading service definition from local FS: /usr/local/hadoop-3.4.0-SNAPSHOT/share/hadoop/yarn/yarn-service-examples/sleeper/sleeper.json 2020-06-17 08:17:35,320 ERROR client.ApiServiceClient: Service name abc is already taken. {code} verifyNoLiveAppInRM only look for appTypes == YarnServiceConstants.APP_TYPE. The correct fix might be adding appTypes == unit-test to the GetApplicationRequest to obtain the correct type of applications. HDFS error message "Dir existing on hdfs." is to safe guard that a instance of the yarn-service application in suspended mode (where there is no copy running in RM), and it's working directory. The error message is not wrong for the suspended use case, and I agree that there might be better way to support --appTypes flag for YARN service API to yield consistent output. Could you refine the patch according it? Thanks > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17137965#comment-17137965 ] Bilwa S T commented on YARN-10310: -- [~eyang] Thanks for explaination. Let me explain my testing steps. I have not deleted file from hdfs in below steps 1. Launch an application with user hdfs using below command two times with same app name {code:java} ./yarn app -launch sleeper-service ../share/hadoop/yarn/yarn-service-examples/sleeper/sleeper.json -appTypes unit-test {code} App launch fails with error saying "Failed to create service sleeper-service, because it already exists." 2. Whereas when i launch an app with hdfs/had...@hadoop.com two times with same app name App launch fails due to DIr existing on hdfs. This behaviour is not same. Please do check once. Thanks > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17137833#comment-17137833 ] Eric Yang commented on YARN-10310: -- [~BilwaST] Thanks for explaining this. If application type is unit-test, and user purposely delete the json file of previous instance of the yarn-service. This would allow second instance of the service to run. YARN allows multiple application submission of the same name, if the application type is unit-test or mapreduce. verifyNoLiveAppInRM only safe guards application type of yarn-service. By using appTypes unit-test, you are triggering unintended approach to launch yarn-service. This is not a bug in YARN service, but how user rigged the system to attempt to trigger unintended code execution path. By shortening the username, it will not prevent verifyNoLiveAppInRM to throw exception for unit-test application type neither. This is working as designed for yarn-service, and allows services and applications co-exist in the same system with different working mode. My recommendation is to submit the app without appTypes to avoid slipping pass verifyNoLiveAppInRM. > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136826#comment-17136826 ] Bilwa S T commented on YARN-10310: -- Hi [~eyang] I ran below command which calls ServiceClient directly. {code:java} ./yarn app -launch sleeper-service ../share/hadoop/yarn/yarn-service-examples/sleeper/sleeper.json -appTypes unit-test {code} I was able to reproduce issue with this command. > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136795#comment-17136795 ] Eric Yang commented on YARN-10310: -- Trunk code without patch 001 produces: Launching application using hdfs/had...@example.com principal: {code} $ kinit hdfs/had...@example.com Password for hdfs/had...@example.com: [hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ ./bin/yarn app -launch rr sleeper 2020-06-16 09:08:28,553 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032 2020-06-16 09:08:29,325 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032 2020-06-16 09:08:29,329 INFO client.ApiServiceClient: Loading service definition from local FS: /usr/local/hadoop-3.4.0-SNAPSHOT/share/hadoop/yarn/yarn-service-examples/sleeper/sleeper.json 2020-06-16 09:08:45,835 INFO client.ApiServiceClient: Application ID: application_1592323643465_0001 [hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ ./bin/hdfs dfs -ls .yarn/services/rr Found 3 items drwxr-x--- - hdfs supergroup 0 2020-06-16 09:08 .yarn/services/rr/conf drwxr-xr-x - hdfs supergroup 0 2020-06-16 09:08 .yarn/services/rr/lib -rw-rw-rw- 1 hdfs supergroup831 2020-06-16 09:08 .yarn/services/rr/rr.json [hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ ./bin/hdfs dfs -rmr .yarn/services/rr rmr: DEPRECATED: Please use '-rm -r' instead. Deleted .yarn/services/rr [hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ ./bin/yarn app -launch rr sleeper 2020-06-16 09:10:18,754 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032 2020-06-16 09:10:19,206 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032 2020-06-16 09:10:19,209 INFO client.ApiServiceClient: Loading service definition from local FS: /usr/local/hadoop-3.4.0-SNAPSHOT/share/hadoop/yarn/yarn-service-examples/sleeper/sleeper.json 2020-06-16 09:10:21,421 ERROR client.ApiServiceClient: Service name rr is already taken. [hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ ./bin/hdfs dfs -ls .yarn/services [hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ klist Ticket cache: FILE:/tmp/krb5cc_123 Default principal: hdfs/had...@example.com Valid starting Expires Service principal 06/16/2020 09:08:15 06/17/2020 09:08:15 krbtgt/example@example.com {code} Launching application using hdfs principal while service file is already deleted from hdfs: {code} $ kinit Password for h...@example.com: [hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ ./bin/yarn app -launch rr sleeper 2020-06-16 09:20:05,737 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032 2020-06-16 09:20:06,405 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at kerberos.example.com/192.168.1.9:8032 2020-06-16 09:20:06,409 INFO client.ApiServiceClient: Loading service definition from local FS: /usr/local/hadoop-3.4.0-SNAPSHOT/share/hadoop/yarn/yarn-service-examples/sleeper/sleeper.json 2020-06-16 09:20:10,082 ERROR client.ApiServiceClient: Service name rr is already taken. [hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ ./bin/hdfs dfs -ls .yarn/services [hdfs@kerberos hadoop-3.4.0-SNAPSHOT]$ {code} If the application is running, verifyNoLiveAppInRM does throw exception. I can not reproduce the claimed issue. I suspect that verifyNoLiveAppInRM did not throw exception due to cluster configuration issues. We should not use getShortUserName() api on the client side. The client must pass the full principal name to server, and only server resolves the short name when necessary. Please check in core-site.xml, the following properties have been configured: {code} hadoop.http.authentication.type kerberos hadoop.http.filter.initializers org.apache.hadoop.security.AuthenticationFilterInitializer {code} If they are not configured correctly, you may be accessing ServiceClient insecurely which result in the errors that you were seeing. > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails >
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136352#comment-17136352 ] Bilwa S T commented on YARN-10310: -- [~eyang] Actually the verifyNoLiveAppInRM check happens before it checks service json from hdfs. I think you can compare the behaviour by submitting with user hdfs and hdfs/had...@example.com. > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136349#comment-17136349 ] Eric Yang commented on YARN-10310: -- [~BilwaST] Without the patch, I was unable to resubmit app with the same name twice. If you deleted service json from hdfs, you are allowed to submit the app again. I think this is working as designed. The check is happening based on data on hdfs rather than what is in resource manager memory. This is safer to prevent data loss in case resource manager crashes. In my system hdfs/had...@example.com mapped to hdfs user principal. It appears that your side didn't. I am not sure how the difference happens, and may need more information. > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136269#comment-17136269 ] Bilwa S T commented on YARN-10310: -- Hi [~eyang] Maybe you can delete file from hdfs and launch an app again. You will be allowed to submit with same name. ie because Service client [send hdfs/had...@example.com|mailto:send%C2%A0hdfs/had...@example.com] as user whereas ClientRMService uses hdfs as user. Whereas with patch , if you launch second app with same name as first app it will throw Failure to create service, maybe cause by existing instance of sleeper service. > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136186#comment-17136186 ] Eric Yang commented on YARN-10310: -- [~BilwaST] I was unable to reproduce the error reported here using hdfs/had...@example.com principal. Failure to create service, maybe cause by existing instance of sleeper service. The service finished running, but it was not destroyed to remove the state file from hdfs. I could not reproduce the described problem, nor patch 001 looks like a solution that would address the described problem. Please clarify. Thanks > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134448#comment-17134448 ] Bilwa S T commented on YARN-10310: -- [~eyang] okay > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134388#comment-17134388 ] Eric Yang commented on YARN-10310: -- [~BilwaST] I am in the process of setting up a new development environment to test this patch. Give me a few days to complete my validations. Thanks > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134382#comment-17134382 ] Bilwa S T commented on YARN-10310: -- Hi [~eyang] please check this when you get free time. Thanks > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130360#comment-17130360 ] Bilwa S T commented on YARN-10310: -- Hi [~eyang] i meant patch is working fine. its working as expected > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130356#comment-17130356 ] Eric Yang commented on YARN-10310: -- It sounds like you need to check auth_to_local rule to make sure that the username is mapped correctly. > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130338#comment-17130338 ] Bilwa S T commented on YARN-10310: -- [~eyang] I tested after applying patch by launching with same name. for second one this exception is thrown. Without patch it never throws this exception > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130321#comment-17130321 ] Eric Yang commented on YARN-10310: -- [~BilwaST] Sorry, I am confused by your comment. Are you saying after applying this patch, hdfs/had...@hadoop.com principal generates "already exists" error message? > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130066#comment-17130066 ] Bilwa S T commented on YARN-10310: -- Testing done for this. > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129454#comment-17129454 ] Bilwa S T commented on YARN-10310: -- Thanks [~eyang] for reviewing. > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128734#comment-17128734 ] Hadoop QA commented on YARN-10310: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 43s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 49s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m 57s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 47s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m 32s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 36s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 75m 29s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/PreCommit-YARN-Build/26132/artifact/out/Dockerfile | | JIRA Issue | YARN-10310 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13005143/YARN-10310.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 7633973971cd 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / fbb87754306 | | Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/26132/testReport/ | | Max. process+thread count | 834 (vs. ulimit of 5500) | | modules |
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128632#comment-17128632 ] Eric Yang commented on YARN-10310: -- Patch 001 looks good, pending Jenkins validation. > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10310) YARN Service - User is able to launch a service with same name
[ https://issues.apache.org/jira/browse/YARN-10310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128488#comment-17128488 ] Bilwa S T commented on YARN-10310: -- cc [~eyang] > YARN Service - User is able to launch a service with same name > -- > > Key: YARN-10310 > URL: https://issues.apache.org/jira/browse/YARN-10310 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10310.001.patch > > > As ServiceClient uses UserGroupInformation.getCurrentUser().getUserName() to > get user whereas ClientRMService#submitApplication uses > UserGroupInformation.getCurrentUser().getShortUserName() to set application > username. > In case of user with name hdfs/had...@hadoop.com. below condition fails > ClientRMService#getApplications() > {code:java} > if (users != null && !users.isEmpty() && > !users.contains(application.getUser())) { > continue; > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org