[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429063#comment-16429063 ] Xiao Chen commented on HADOOP-12253: Per Manoj's comment above, I'm resolving this as a dup of HADOOP-13442. Please feel free to reopen if this isn't true. > ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0 > > > Key: HADOOP-12253 > URL: https://issues.apache.org/jira/browse/HADOOP-12253 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0 hive 1.1.0 tez0.7 cenos6.4 >Reporter: JiJun Tang >Assignee: Ajith S >Priority: Major > Attachments: HADOOP-12253.2.patch, HADOOP-12253.patch > > > When I enable hdfs federation.I run a query on hive on tez. Then it occur a > exception: > {noformat} > 8.784 PM WARNorg.apache.hadoop.security.UserGroupInformation No > groups available for user tangjijun > 3:12:28.784 PMERROR org.apache.hadoop.hive.ql.exec.Task Failed > to execute tez graph. > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.getFileStatus(ViewFileSystem.java:771) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:359) > at > org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:955) > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:184) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:787) > at org.apache.tez.client.TezClient.start(TezClient.java:337) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:234) > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1640) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1399) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:144) > at > org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69) > at > org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at > org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I digging into the issue,I found the code snippet in ViewFileSystem.java as > follows: > {noformat} > @Override > public FileStatus getFileStatus(Path f) throws IOException { > checkPathIsSlash(f); > return new FileStatus(0, true, 0, 0, creationTime, creationTime, > PERMISSION_555, ugi.getUserName(), ugi.getGroupNames()[0], > new Path(theInternalDir.fullPath).makeQualified( > myUri, ROOT_PATH)); > } > {noformat} > If the node in cluster haven't creat user like > tangjijun,ugi.getGroupNames()[0] will throw > ArrayIndexOutOfBoundsException.Because no user mean no group. > I create user tangjijun on that node. Then the job was executed normally. > I think this code should check ugi.getGroupNames() is empty.When it is empty > ,then print some log. Not to throw exception. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15537633#comment-15537633 ] Hadoop QA commented on HADOOP-12253: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color} | {color:red} HADOOP-12253 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | HADOOP-12253 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12795761/HADOOP-12253.2.patch | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/10637/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0 > > > Key: HADOOP-12253 > URL: https://issues.apache.org/jira/browse/HADOOP-12253 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0 hive 1.1.0 tez0.7 cenos6.4 >Reporter: JiJun Tang >Assignee: Ajith S > Attachments: HADOOP-12253.2.patch, HADOOP-12253.patch > > > When I enable hdfs federation.I run a query on hive on tez. Then it occur a > exception: > {noformat} > 8.784 PM WARNorg.apache.hadoop.security.UserGroupInformation No > groups available for user tangjijun > 3:12:28.784 PMERROR org.apache.hadoop.hive.ql.exec.Task Failed > to execute tez graph. > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.getFileStatus(ViewFileSystem.java:771) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:359) > at > org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:955) > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:184) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:787) > at org.apache.tez.client.TezClient.start(TezClient.java:337) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:234) > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1640) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1399) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:144) > at > org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69) > at > org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at > org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I digging into the issue,I found the code snippet in ViewFileSystem.java as > follows: > {noformat} > @Override > public FileStatus getFileStatus(Path f) throws IOException { > checkPathIsSlash(f); > return new FileStatus(0, true, 0, 0, creationTime, creationTime, > PERMISSION_555, ugi.getUserName(), ugi.getGroupNames()[0], > new Path(theInternalDir.fullPath).makeQualified( > myUri, ROOT_PATH)); > } > {noformat} > If the node in cluster haven't creat user like > tangjijun,ugi.getGroupNames()[0] will throw > ArrayIn
[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15537504#comment-15537504 ] Manoj Govindassamy commented on HADOOP-12253: - Hi [~ajithshetty], Looks like the recent fix made by HADOOP-13442 also solves the core issue in parsing UGI. Will that solve this issue HADOOP-12253 also ? {code} @Override public FileStatus getFileStatus(Path f) throws IOException { checkPathIsSlash(f); return new FileStatus(0, true, 0, 0, creationTime, creationTime, PERMISSION_555, ugi.getUserName(), ugi.getPrimaryGroupName(), new Path(theInternalDir.fullPath).makeQualified( myUri, ROOT_PATH)); } {code} > ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0 > > > Key: HADOOP-12253 > URL: https://issues.apache.org/jira/browse/HADOOP-12253 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0 hive 1.1.0 tez0.7 cenos6.4 >Reporter: JiJun Tang >Assignee: Ajith S > Attachments: HADOOP-12253.2.patch, HADOOP-12253.patch > > > When I enable hdfs federation.I run a query on hive on tez. Then it occur a > exception: > {noformat} > 8.784 PM WARNorg.apache.hadoop.security.UserGroupInformation No > groups available for user tangjijun > 3:12:28.784 PMERROR org.apache.hadoop.hive.ql.exec.Task Failed > to execute tez graph. > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.getFileStatus(ViewFileSystem.java:771) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:359) > at > org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:955) > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:184) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:787) > at org.apache.tez.client.TezClient.start(TezClient.java:337) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:234) > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1640) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1399) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:144) > at > org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69) > at > org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at > org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I digging into the issue,I found the code snippet in ViewFileSystem.java as > follows: > {noformat} > @Override > public FileStatus getFileStatus(Path f) throws IOException { > checkPathIsSlash(f); > return new FileStatus(0, true, 0, 0, creationTime, creationTime, > PERMISSION_555, ugi.getUserName(), ugi.getGroupNames()[0], > new Path(theInternalDir.fullPath).makeQualified( > myUri, ROOT_PATH)); > } > {noformat} > If the node in cluster haven't creat user like > tangjijun,ugi.getGroupNames()[0] will throw > ArrayIndexOutOfBoundsException.Because no user mean no group. > I create user tangjijun on that node. Then the job was executed normally. > I think this code should check ugi.getGroupNames() is empty.When it is empty > ,then print some log. Not to throw exception. -- This message was sent by A
[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15215417#comment-15215417 ] Hadoop QA commented on HADOOP-12253: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 44s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 54s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 37s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 34s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s {color} | {color:green} trunk passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 40s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 41s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 41s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 32s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 47s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s {color} | {color:green} the patch passed with JDK v1.8.0_74 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 16s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_74. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 33s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_95. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 24s {color} | {color:red} Patch generated 3 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 56m 51s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_74 Timed out junit tests | org.apache.hadoop.util.TestNativeLibraryChecker | | JDK v1.7.0_95 Timed out junit tests | org.apache.hadoop.util.TestNativeLibraryChecker | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:fbe3e86 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12795761/HADOOP-12253.2.patch | | JIRA Issue | HADOOP-12253 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 3e57cb7e083a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personal
[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089491#comment-15089491 ] Hadoop QA commented on HADOOP-12253: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 53s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 35s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 15s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 56s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 39s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 32s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 32s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 16s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 16s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s {color} | {color:red} Patch generated 3 new checkstyle issues in hadoop-common-project/hadoop-common (total was 64, now 67). {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 3s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 41s {color} | {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 50s {color} | {color:red} hadoop-common in the patch failed with JDK v1.7.0_91. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 73m 41s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_66 Failed junit tests | hadoop.metrics2.impl.TestGangliaMetrics | | JDK v1.7.0_91 Failed junit tests | hadoop.fs.shell.TestCopyPreserveFlag | | | hadoop.ipc.TestIPC | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:0ca8df7 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12749752/HADOOP-12253.patch | | JIRA Issue | HADOOP-12253 | | Optional Tests | asflicense compile javac javadoc mvni
[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089345#comment-15089345 ] Steve Loughran commented on HADOOP-12253: - Looking at this code, {{ugi.getGroupNames().length < 1 ? null : ugi.getGroupNames()[0];}} re-occurs in 3 places. It should be factored out into some method called "getFirstGroup(ugi)". Make it static and you can then add the test that this patch needs > ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0 > > > Key: HADOOP-12253 > URL: https://issues.apache.org/jira/browse/HADOOP-12253 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0 hive 1.1.0 tez0.7 cenos6.4 >Reporter: tangjunjie >Assignee: Ajith S > Attachments: HADOOP-12253.patch > > > When I enable hdfs federation.I run a query on hive on tez. Then it occur a > exception: > {noformat} > 8.784 PM WARNorg.apache.hadoop.security.UserGroupInformation No > groups available for user tangjijun > 3:12:28.784 PMERROR org.apache.hadoop.hive.ql.exec.Task Failed > to execute tez graph. > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.getFileStatus(ViewFileSystem.java:771) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:359) > at > org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:955) > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:184) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:787) > at org.apache.tez.client.TezClient.start(TezClient.java:337) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:234) > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1640) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1399) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:144) > at > org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69) > at > org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at > org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I digging into the issue,I found the code snippet in ViewFileSystem.java as > follows: > {noformat} > @Override > public FileStatus getFileStatus(Path f) throws IOException { > checkPathIsSlash(f); > return new FileStatus(0, true, 0, 0, creationTime, creationTime, > PERMISSION_555, ugi.getUserName(), ugi.getGroupNames()[0], > new Path(theInternalDir.fullPath).makeQualified( > myUri, ROOT_PATH)); > } > {noformat} > If the node in cluster haven't creat user like > tangjijun,ugi.getGroupNames()[0] will throw > ArrayIndexOutOfBoundsException.Because no user mean no group. > I create user tangjijun on that node. Then the job was executed normally. > I think this code should check ugi.getGroupNames() is empty.When it is empty > ,then print some log. Not to throw exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14681265#comment-14681265 ] Hadoop QA commented on HADOOP-12253: \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 23m 2s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | | {color:green}+1{color} | javac | 13m 15s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 13m 12s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 52s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 1m 24s | The applied patch generated 3 new checkstyle issues (total was 64, now 67). | | {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 44s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 43s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 2m 34s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:red}-1{color} | common tests | 25m 19s | Tests failed in hadoop-common. | | | | 82m 8s | | \\ \\ || Reason || Tests || | Failed unit tests | hadoop.net.TestNetUtils | | | hadoop.ha.TestZKFailoverController | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12749752/HADOOP-12253.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / fa1d84a | | checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/7435/artifact/patchprocess/diffcheckstylehadoop-common.txt | | whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/7435/artifact/patchprocess/whitespace.txt | | hadoop-common test log | https://builds.apache.org/job/PreCommit-HADOOP-Build/7435/artifact/patchprocess/testrun_hadoop-common.txt | | Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/7435/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/7435/console | This message was automatically generated. > ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0 > > > Key: HADOOP-12253 > URL: https://issues.apache.org/jira/browse/HADOOP-12253 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0 hive 1.1.0 tez0.7 cenos6.4 >Reporter: tangjunjie >Assignee: Ajith S > Attachments: HADOOP-12253.patch > > > When I enable hdfs federation.I run a query on hive on tez. Then it occur a > exception: > {noformat} > 8.784 PM WARNorg.apache.hadoop.security.UserGroupInformation No > groups available for user tangjijun > 3:12:28.784 PMERROR org.apache.hadoop.hive.ql.exec.Task Failed > to execute tez graph. > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.getFileStatus(ViewFileSystem.java:771) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:359) > at > org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:955) > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:184) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:787) > at org.apache.tez.client.TezClient.start(TezClient.java:337) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:234) > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1640) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1399) > at org.apache.hadoo
[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14653190#comment-14653190 ] Ajith S commented on HADOOP-12253: -- Hi {noformat}ugi.getGroupNames()[0]{noformat} seems to cause trouble in other command like listStatus or getAclStatus. So as a fix, can we check if user belongs to at least one group when ViewFileSystem is initialized i.e in ViewFileSystem() constructor.?? so that issue is reflected in setup stage.?? > ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0 > > > Key: HADOOP-12253 > URL: https://issues.apache.org/jira/browse/HADOOP-12253 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0 hive 1.1.0 tez0.7 cenos6.4 >Reporter: tangjunjie >Assignee: Ajith S > > When I enable hdfs federation.I run a query on hive on tez. Then it occur a > exception: > {noformat} > 8.784 PM WARNorg.apache.hadoop.security.UserGroupInformation No > groups available for user tangjijun > 3:12:28.784 PMERROR org.apache.hadoop.hive.ql.exec.Task Failed > to execute tez graph. > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.getFileStatus(ViewFileSystem.java:771) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:359) > at > org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:955) > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:184) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:787) > at org.apache.tez.client.TezClient.start(TezClient.java:337) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:234) > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1640) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1399) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:144) > at > org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69) > at > org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at > org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I digging into the issue,I found the code snippet in ViewFileSystem.java as > follows: > {noformat} > @Override > public FileStatus getFileStatus(Path f) throws IOException { > checkPathIsSlash(f); > return new FileStatus(0, true, 0, 0, creationTime, creationTime, > PERMISSION_555, ugi.getUserName(), ugi.getGroupNames()[0], > new Path(theInternalDir.fullPath).makeQualified( > myUri, ROOT_PATH)); > } > {noformat} > If the node in cluster haven't creat user like > tangjijun,ugi.getGroupNames()[0] will throw > ArrayIndexOutOfBoundsException.Because no user mean no group. > I create user tangjijun on that node. Then the job was executed normally. > I think this code should check ugi.getGroupNames() is empty.When it is empty > ,then print some log. Not to throw exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637980#comment-14637980 ] tangjunjie commented on HADOOP-12253: - This issue recall me to another issus I fired . https://issues.apache.org/jira/browse/HDFS-8663 . No user found on namenode server will cause sys cpu high. > ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0 > > > Key: HADOOP-12253 > URL: https://issues.apache.org/jira/browse/HADOOP-12253 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0 hive 1.1.0 tez0.7 cenos6.4 >Reporter: tangjunjie >Assignee: Ajith S > > When I enable hdfs federation.I run a query on hive on tez. Then it occur a > exception: > {noformat} > 8.784 PM WARNorg.apache.hadoop.security.UserGroupInformation No > groups available for user tangjijun > 3:12:28.784 PMERROR org.apache.hadoop.hive.ql.exec.Task Failed > to execute tez graph. > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.getFileStatus(ViewFileSystem.java:771) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:359) > at > org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:955) > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:184) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:787) > at org.apache.tez.client.TezClient.start(TezClient.java:337) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:234) > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1640) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1399) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:144) > at > org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69) > at > org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at > org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I digging into the issue,I found the code snippet in ViewFileSystem.java as > follows: > {noformat} > @Override > public FileStatus getFileStatus(Path f) throws IOException { > checkPathIsSlash(f); > return new FileStatus(0, true, 0, 0, creationTime, creationTime, > PERMISSION_555, ugi.getUserName(), ugi.getGroupNames()[0], > new Path(theInternalDir.fullPath).makeQualified( > myUri, ROOT_PATH)); > } > {noformat} > If the node in cluster haven't creat user like > tangjijun,ugi.getGroupNames()[0] will throw > ArrayIndexOutOfBoundsException.Because no user mean no group. > I create user tangjijun on that node. Then the job was executed normally. > I think this code should check ugi.getGroupNames() is empty.When it is empty > ,then print some log. Not to throw exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637127#comment-14637127 ] zhihai xu commented on HADOOP-12253: Also {{Groups#getGroups}} will call {{Groups#fetchGroupList}} which will call {{ShellBasedUnixGroupsMapping#getUnixGroups}} if you use the default value for "hadoop.security.group.mapping". {{ShellBasedUnixGroupsMapping#getUnixGroups}} will run shell coomand {{Shell.getGroupsForUserCommand(user)}} to get the group information for the user. {code} public static String[] getGroupsForUserCommand(final String user) { //'groups username' command return is non-consistent across different unixes return (WINDOWS)? new String[] { WINUTILS, "groups", "-F", "\"" + user + "\""} : new String [] {"bash", "-c", "id -gn " + user + "&& id -Gn " + user}; } {code} For linux, it uses the following command: {code} id -gn && id -Gn {code} So you can run the above command to verify your environment. > ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0 > > > Key: HADOOP-12253 > URL: https://issues.apache.org/jira/browse/HADOOP-12253 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0 hive 1.1.0 tez0.7 cenos6.4 >Reporter: tangjunjie >Assignee: Ajith S > > When I enable hdfs federation.I run a query on hive on tez. Then it occur a > exception: > {noformat} > 8.784 PM WARNorg.apache.hadoop.security.UserGroupInformation No > groups available for user tangjijun > 3:12:28.784 PMERROR org.apache.hadoop.hive.ql.exec.Task Failed > to execute tez graph. > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.getFileStatus(ViewFileSystem.java:771) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:359) > at > org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:955) > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:184) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:787) > at org.apache.tez.client.TezClient.start(TezClient.java:337) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:234) > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1640) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1399) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:144) > at > org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69) > at > org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at > org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I digging into the issue,I found the code snippet in ViewFileSystem.java as > follows: > {noformat} > @Override > public FileStatus getFileStatus(Path f) throws IOException { > checkPathIsSlash(f); > return new FileStatus(0, true, 0, 0, creationTime, creationTime, > PERMISSION_555, ugi.getUserName(), ugi.getGroupNames()[0], > new Path(theInternalDir.fullPath).makeQualified( > myUri, ROOT_PATH)); > } > {noformat} > If the node in cluster haven't creat user like > tangjijun,ugi.getGroupNames()[0] will throw > ArrayIndexOutOfBoundsException.Because no
[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14636467#comment-14636467 ] zhihai xu commented on HADOOP-12253: yes, it looks like there is a problem to get the group name of the user. The waning message "No groups available for user tangjijun" is printed at the following code {{UserGroupInformation#getGroupNames}}: {code} public synchronized String[] getGroupNames() { ensureInitialized(); try { Set result = new LinkedHashSet (groups.getGroups(getShortUserName())); return result.toArray(new String[result.size()]); } catch (IOException ie) { LOG.warn("No groups available for user " + getShortUserName()); return new String[0]; } } {code} So the IOException happened in {{Groups#getGroups}} And ArrayIndexOutOfBoundsException is from ugi.getGroupNames()[0] due to zero size string array returned from {{getGroupNames}} > ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0 > > > Key: HADOOP-12253 > URL: https://issues.apache.org/jira/browse/HADOOP-12253 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0 hive 1.1.0 tez0.7 cenos6.4 >Reporter: tangjunjie >Assignee: Ajith S > > When I enable hdfs federation.I run a query on hive on tez. Then it occur a > exception: > {noformat} > 8.784 PM WARNorg.apache.hadoop.security.UserGroupInformation No > groups available for user tangjijun > 3:12:28.784 PMERROR org.apache.hadoop.hive.ql.exec.Task Failed > to execute tez graph. > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.getFileStatus(ViewFileSystem.java:771) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:359) > at > org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:955) > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:184) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:787) > at org.apache.tez.client.TezClient.start(TezClient.java:337) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:234) > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1640) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1399) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:144) > at > org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69) > at > org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at > org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I digging into the issue,I found the code snippet in ViewFileSystem.java as > follows: > {noformat} > @Override > public FileStatus getFileStatus(Path f) throws IOException { > checkPathIsSlash(f); > return new FileStatus(0, true, 0, 0, creationTime, creationTime, > PERMISSION_555, ugi.getUserName(), ugi.getGroupNames()[0], > new Path(theInternalDir.fullPath).makeQualified( > myUri, ROOT_PATH)); > } > {noformat} > If the node in cluster haven't creat user like > tangjijun,ugi.getGroupNames()[0] will throw > ArrayIndexOutOfBoundsException.Because no user mean no group. > I create user tangjijun on that node. Then the job was executed normally. >
[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14636409#comment-14636409 ] tangjunjie commented on HADOOP-12253: - Regarding to the environment, do you mean that I have to create the using user on every node of the cluster? > ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0 > > > Key: HADOOP-12253 > URL: https://issues.apache.org/jira/browse/HADOOP-12253 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0 hive 1.1.0 tez0.7 cenos6.4 >Reporter: tangjunjie >Assignee: Ajith S > > When I enable hdfs federation.I run a query on hive on tez. Then it occur a > exception: > {noformat} > 8.784 PM WARNorg.apache.hadoop.security.UserGroupInformation No > groups available for user tangjijun > 3:12:28.784 PMERROR org.apache.hadoop.hive.ql.exec.Task Failed > to execute tez graph. > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.getFileStatus(ViewFileSystem.java:771) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:359) > at > org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:955) > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:184) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:787) > at org.apache.tez.client.TezClient.start(TezClient.java:337) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:234) > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1640) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1399) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:144) > at > org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69) > at > org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at > org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > I digging into the issue,I found the code snippet in ViewFileSystem.java as > follows: > {noformat} > @Override > public FileStatus getFileStatus(Path f) throws IOException { > checkPathIsSlash(f); > return new FileStatus(0, true, 0, 0, creationTime, creationTime, > PERMISSION_555, ugi.getUserName(), ugi.getGroupNames()[0], > new Path(theInternalDir.fullPath).makeQualified( > myUri, ROOT_PATH)); > } > {noformat} > If the node in cluster haven't creat user like > tangjijun,ugi.getGroupNames()[0] will throw > ArrayIndexOutOfBoundsException.Because no user mean no group. > I create user tangjijun on that node. Then the job was executed normally. > I think this code should check ugi.getGroupNames() is empty.When it is empty > ,then print some log. Not to throw exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14636389#comment-14636389 ] Yong Zhang commented on HADOOP-12253: - sorry, I mean not hadoop issue. > ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0 > > > Key: HADOOP-12253 > URL: https://issues.apache.org/jira/browse/HADOOP-12253 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0 hive 1.1.0 tez0.7 cenos6.4 >Reporter: tangjunjie >Assignee: Ajith S > > When I enable hdfs federation.I run a query on hive on tez. Then it occur a > exception: > 8.784 PM WARNorg.apache.hadoop.security.UserGroupInformation No > groups available for user tangjijun > 3:12:28.784 PMERROR org.apache.hadoop.hive.ql.exec.Task > Failed to execute tez graph. > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.getFileStatus(ViewFileSystem.java:771) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:359) > at > org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:955) > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:184) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:787) > at org.apache.tez.client.TezClient.start(TezClient.java:337) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:234) > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1640) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1399) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:144) > at > org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69) > at > org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at > org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > I digging into the issue,I found the code snippet in ViewFileSystem.java as > follows: > @Override > public FileStatus getFileStatus(Path f) throws IOException { > checkPathIsSlash(f); > return new FileStatus(0, true, 0, 0, creationTime, creationTime, > PERMISSION_555, ugi.getUserName(), ugi.getGroupNames()[0], > new Path(theInternalDir.fullPath).makeQualified( > myUri, ROOT_PATH)); > } > If the node in cluster haven't creat user like > tangjijun,ugi.getGroupNames()[0] will throw > ArrayIndexOutOfBoundsException.Because no user mean no group. > I create user tangjijun on that node. Then the job was executed normally. > I think this code should check ugi.getGroupNames() is empty.When it is empty > ,then print some log. Not to throw exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HADOOP-12253) ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0
[ https://issues.apache.org/jira/browse/HADOOP-12253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14636388#comment-14636388 ] Yong Zhang commented on HADOOP-12253: - I do not think it is HDFS issue, you must make sure your environment is fine when you run anything. > ViewFileSystem getFileStatus java.lang.ArrayIndexOutOfBoundsException: 0 > > > Key: HADOOP-12253 > URL: https://issues.apache.org/jira/browse/HADOOP-12253 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.6.0 > Environment: hadoop 2.6.0 hive 1.1.0 tez0.7 cenos6.4 >Reporter: tangjunjie >Assignee: Ajith S > > When I enable hdfs federation.I run a query on hive on tez. Then it occur a > exception: > 8.784 PM WARNorg.apache.hadoop.security.UserGroupInformation No > groups available for user tangjijun > 3:12:28.784 PMERROR org.apache.hadoop.hive.ql.exec.Task > Failed to execute tez graph. > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.fs.viewfs.ViewFileSystem$InternalDirOfViewFs.getFileStatus(ViewFileSystem.java:771) > at > org.apache.hadoop.fs.viewfs.ViewFileSystem.getFileStatus(ViewFileSystem.java:359) > at > org.apache.tez.client.TezClientUtils.checkAncestorPermissionsForAllUsers(TezClientUtils.java:955) > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:184) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:787) > at org.apache.tez.client.TezClient.start(TezClient.java:337) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:191) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.updateSession(TezTask.java:234) > at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1640) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1399) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044) > at > org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:144) > at > org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69) > at > org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at > org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > I digging into the issue,I found the code snippet in ViewFileSystem.java as > follows: > @Override > public FileStatus getFileStatus(Path f) throws IOException { > checkPathIsSlash(f); > return new FileStatus(0, true, 0, 0, creationTime, creationTime, > PERMISSION_555, ugi.getUserName(), ugi.getGroupNames()[0], > new Path(theInternalDir.fullPath).makeQualified( > myUri, ROOT_PATH)); > } > If the node in cluster haven't creat user like > tangjijun,ugi.getGroupNames()[0] will throw > ArrayIndexOutOfBoundsException.Because no user mean no group. > I create user tangjijun on that node. Then the job was executed normally. > I think this code should check ugi.getGroupNames() is empty.When it is empty > ,then print some log. Not to throw exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)