[ https://issues.apache.org/jira/browse/YARN-9511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949890#comment-16949890 ]
liusheng edited comment on YARN-9511 at 10/12/19 3:49 AM: ---------------------------------------------------------- Hi [~snemeth] [~adam.antal], Thank you both for care about this issue. I have take some time tried to find the reason of this issue. This issue will effect the tests of *TestAuxServices*, and will cause *2 Errors 9 Failures*, see: {code:java} Failures: TestAuxServices.testAuxServiceRecoverySetup:717 expected:<2> but was:<0> TestAuxServices.testAuxServicesManifestPermissions:874 expected:<2> but was:<0> TestAuxServices.testAuxServicesMeta:638 Invalid mix of services expected:<6> but was:<1> TestAuxServices.testAuxServices:610 Invalid mix of services expected:<6> but was:<1> TestAuxServices.testCustomizedAuxServiceClassPath:416 TestAuxServices.testManualReload:919 expected:<2> but was:<0> TestAuxServices.testRemoteAuxServiceClassPath:313 The permission of the jar is wrong.Should throw out exception. TestAuxServices.testRemoveManifest:897 expected:<2> but was:<0> TestAuxServices.testValidAuxServiceName:698 Should receive the exception. Errors: TestAuxServices.testAuxUnexpectedStop:664 » NoSuchElement TestAuxServices.testRemoteAuxServiceClassPath:334 » YarnRuntime The remote jar... {code} After debuging, I found all these issues are directly or indirectly related with files permissions. there are two situations when running these tests: # *useManifest* enabled, when running tests with useManifest enabled, the tests will check and use the manifest file: {code:java} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/test-dir/TestAuxServices/manifest.txt{code} this file and all its parents directories must not be writable by group or others, see: {code:java} private boolean checkManifestPermissions(FileStatus status) throws IOException { if ((status.getPermission().toShort() & 0022) != 0) { LOG.error("Manifest file and parents must not be writable by group or " + "others. The current Permission of " + status.getPath() + " is " + status.getPermission()); return false; } Path parent = status.getPath().getParent(); if (parent == null) { return true; } return checkManifestPermissions(manifestFS.getFileStatus(parent)); }{code} ** # *useManifest not* enabled, when running tests with useManifest enabled, tests will use a *test-runjar.jar* file {code:java} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/test-dir/TestAuxServices/test-runjar.jar {code} related code for checking its permission: {code:java} private Path maybeDownloadJars(String sName, String className, String remoteFile, AuxServiceFile.TypeEnum type, Configuration conf) throws IOException { // load AuxiliaryService from remote classpath FileContext localLFS = getLocalFileContext(conf); // create NM aux-service dir in NM localdir if it does not exist. Path nmAuxDir = dirsHandler.getLocalPathForWrite("." + Path.SEPARATOR + NM_AUX_SERVICE_DIR); if (!localLFS.util().exists(nmAuxDir)) { try { localLFS.mkdir(nmAuxDir, NM_AUX_SERVICE_DIR_PERM, true); } catch (IOException ex) { throw new YarnRuntimeException("Fail to create dir:" + nmAuxDir.toString(), ex); } } Path src = new Path(remoteFile); FileContext remoteLFS = getRemoteFileContext(src.toUri(), conf); FileStatus scFileStatus = remoteLFS.getFileStatus(src); if (!scFileStatus.getOwner().equals( this.userUGI.getShortUserName())) { throw new YarnRuntimeException("The remote jarfile owner:" + scFileStatus.getOwner() + " is not the same as the NM user:" + this.userUGI.getShortUserName() + "."); } if ((scFileStatus.getPermission().toShort() & 0022) != 0) { throw new YarnRuntimeException("The remote jarfile should not " + "be writable by group or others. " + "The current Permission is " + scFileStatus.getPermission().toShort()); } {code} According to the above reasons, I have tried to change *manifest.txt* file parents directories without *writeable* permission of group and others. and change the *umask to 077*, which will effect new created file and directories permissions, because the *manifest.txt* and *run-tests.jar* will be new created when running tests. {code:java} chmod go-w yourpath/hadoop/ -R umask 022 umask {code} After doing above and re-run tests of *TestAuxServices*, all the tests can pass. Actually, I am a new comer to Hadoop, so I am not sure whether this is a bug of Hadoop or not. could you please give some suggestions? Thanks. was (Author: seanlau): Hi [~snemeth] [~adam.antal], Thank you both for care about this issue. I have take some time tried to find the reason of this issue. This issue will effect the tests of *TestAuxServices*, and will cause *2 Errors 9 Failures*, see: {code:java} Failures: TestAuxServices.testAuxServiceRecoverySetup:717 expected:<2> but was:<0> TestAuxServices.testAuxServicesManifestPermissions:874 expected:<2> but was:<0> TestAuxServices.testAuxServicesMeta:638 Invalid mix of services expected:<6> but was:<1> TestAuxServices.testAuxServices:610 Invalid mix of services expected:<6> but was:<1> TestAuxServices.testCustomizedAuxServiceClassPath:416 TestAuxServices.testManualReload:919 expected:<2> but was:<0> TestAuxServices.testRemoteAuxServiceClassPath:313 The permission of the jar is wrong.Should throw out exception. TestAuxServices.testRemoveManifest:897 expected:<2> but was:<0> TestAuxServices.testValidAuxServiceName:698 Should receive the exception. Errors: TestAuxServices.testAuxUnexpectedStop:664 » NoSuchElement TestAuxServices.testRemoteAuxServiceClassPath:334 » YarnRuntime The remote jar... {code} After debuging, I found all these issues are directly or indirectly related with files permissions. there are two situations when running these tests: # *useManifest* enabled, when running tests with useManifest enabled, the tests will check and use the manifest file: {code:java} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/test-dir/TestAuxServices/manifest.txt{code} this file and all its parents directories must not be writable by group or others, see: {code:java} private boolean checkManifestPermissions(FileStatus status) throws IOException { if ((status.getPermission().toShort() & 0022) != 0) { LOG.error("Manifest file and parents must not be writable by group or " + "others. The current Permission of " + status.getPath() + " is " + status.getPermission()); return false; } Path parent = status.getPath().getParent(); if (parent == null) { return true; } return checkManifestPermissions(manifestFS.getFileStatus(parent)); }{code} # *useManifest not* enabled, when running tests with useManifest enabled, tests will use a test-jar.jar file {code:java} private Path maybeDownloadJars(String sName, String className, String remoteFile, AuxServiceFile.TypeEnum type, Configuration conf) throws IOException { // load AuxiliaryService from remote classpath FileContext localLFS = getLocalFileContext(conf); // create NM aux-service dir in NM localdir if it does not exist. Path nmAuxDir = dirsHandler.getLocalPathForWrite("." + Path.SEPARATOR + NM_AUX_SERVICE_DIR); if (!localLFS.util().exists(nmAuxDir)) { try { localLFS.mkdir(nmAuxDir, NM_AUX_SERVICE_DIR_PERM, true); } catch (IOException ex) { throw new YarnRuntimeException("Fail to create dir:" + nmAuxDir.toString(), ex); } } Path src = new Path(remoteFile); FileContext remoteLFS = getRemoteFileContext(src.toUri(), conf); FileStatus scFileStatus = remoteLFS.getFileStatus(src); if (!scFileStatus.getOwner().equals( this.userUGI.getShortUserName())) { throw new YarnRuntimeException("The remote jarfile owner:" + scFileStatus.getOwner() + " is not the same as the NM user:" + this.userUGI.getShortUserName() + "."); } if ((scFileStatus.getPermission().toShort() & 0022) != 0) { throw new YarnRuntimeException("The remote jarfile should not " + "be writable by group or others. " + "The current Permission is " + scFileStatus.getPermission().toShort()); } {code} According to the above reasons, I have tried to change *manifest.txt* file parents directories without *writeable* permission of group and others. and change the *umask to 077*, which will effect new created file and directories permissions, because the *manifest.txt* and *run-tests.jar* will be new created when running tests. {code:java} chmod go-w yourpath/hadoop/ -R umask 022 umask {code} After doing above and re-run tests of *TestAuxServices*, all the tests can pass. Actually, I am a new comer to Hadoop, so I am not sure whether this is a bug of Hadoop or not. could you please give some suggestions? Thanks. > [JDK11] TestAuxServices#testRemoteAuxServiceClassPath YarnRuntimeException: > The remote jarfile should not be writable by group or others. The current > Permission is 436 > ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > Key: YARN-9511 > URL: https://issues.apache.org/jira/browse/YARN-9511 > Project: Hadoop YARN > Issue Type: Bug > Components: test > Reporter: Siyao Meng > Assignee: Szilard Nemeth > Priority: Major > > Found in maven JDK 11 unit test run. Compiled on JDK 8. > {code} > [ERROR] > testRemoteAuxServiceClassPath(org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices) > Time elapsed: 0.551 s <<< > ERROR!org.apache.hadoop.yarn.exceptions.YarnRuntimeException: The remote > jarfile should not be writable by group or others. The current Permission is > 436 > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:202) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices.testRemoteAuxServiceClassPath(TestAuxServices.java:268) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org