[jira] [Created] (MAPREDUCE-5477) In JHS wrong values coming for following fields "Average Reduce Time,Average Shuffle Time,Average Merge Time"
Nishan Shetty created MAPREDUCE-5477:
Summary: In JHS wrong values coming for following fields "Average
Reduce Time,Average Shuffle Time,Average Merge Time"
Key: MAPREDUCE-5477
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5477
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Nishan Shetty
If multiple app master attempts are there wrong values are coming for following
fields
{noformat}
Average Reduce Time 381366hrs, 44mins, 49sec
Average Shuffle Time -381366hrs, -44mins, -45sec
Average Merge Time -381366hrs, -44mins, -45sec
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Hadoop-Mapreduce-trunk - Build # 1526 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1526/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 31769 lines...] at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782) at hudson.model.Build$BuildExecution.build(Build.java:199) at hudson.model.Build$BuildExecution.doRun(Build.java:160) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:567) at hudson.model.Run.execute(Run.java:1603) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:247) Caused by: hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:516) at hudson.remoting.Request.call(Request.java:129) at hudson.remoting.Channel.call(Channel.java:714) at hudson.FilePath.act(FilePath.java:898) ... 13 more Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1316) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:41) at hudson.remoting.RequestAbortedException.wrapForRethrow(RequestAbortedException.java:34) at hudson.remoting.Request.call(Request.java:174) at hudson.remoting.Channel.call(Channel.java:714) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:167) at com.sun.proxy.$Proxy44.join(Unknown Source) at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:925) at hudson.Launcher$ProcStarter.join(Launcher.java:360) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91) at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:782) at hudson.model.Build$BuildExecution.build(Build.java:199) at hudson.model.Build$BuildExecution.doRun(Build.java:160) at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:567) at hudson.model.Run.execute(Run.java:1603) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:247) Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:299) at hudson.remoting.Channel.terminate(Channel.java:774) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69) Caused by: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50) Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2596) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1316) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:71) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) ### ## FAILED TESTS (if any) ## No tests ran.
Re: [VOTE] Release Apache Hadoop 2.0.6-alpha
+1 non-binding verified signatures and checksum built source single node cluster tests Rob On 08/11/2013 12:46 AM, Konstantin Boudnik wrote: All, I have created a release candidate (rc0) for hadoop-2.0.6-alpha that I would like to release. This is a stabilization release that includes fixed for a couple a of issues as outlined on the security list. The RC is available at: http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc0/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc0 The maven artifacts are available via repository.apache.org. Please try the release bits and vote; the vote will run for the usual 7 days. Thanks for your voting Cos
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
On Wed, Aug 21, 2013 at 3:49 PM, Stack wrote: > On Wed, Aug 21, 2013 at 1:25 PM, Colin McCabe wrote: > >> St.Ack wrote: >> >> > + Once I figured where the logs were, found that JAVA_HOME was not being >> > exported (don't need this in hadoop-2.0.5 for instance). Adding an >> > exported JAVA_HOME to my running shell which don't seem right but it took >> > care of it (I gave up pretty quick on messing w/ >> > yarn.nodemanager.env-whitelist and yarn.nodemanager.admin-env -- I wasn't >> > getting anywhere) >> >> I thought that we were always supposed to have JAVA_HOME set when >> running any of these commands. At least, I do. How else can the >> system disambiguate between different Java installs? I need 2 >> installs to test with JDK7. >> >> > > That is fair enough but I did not need to define this explicitly previously > (for hadoop-2.0.5-alpha for instance) or the JAVA_HOME that was figured in > start scripts was propagated and now is not (I have not dug in). > > > >> > + This did not seem to work for me: >> > hadoop.security.group.mapping >> > >> org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback> > lue>. >> >> We've seen this before. I think your problem is that you have >> java.library.path set correctly (what System.loadLibrary checks), but >> your system library path does not include a necessary dependency of >> libhadoop.so-- most likely, libjvm.so. Probably, we should fix >> NativeCodeLoader to actually make a function call in libhadoop.so >> before it declares everything OK. >> > > My expectation was that if native group lookup fails, as it does here, then > the 'Fallback' would kick in and we'd do the Shell query. This mechanism > does not seem to be working. I filed https://issues.apache.org/jira/browse/HADOOP-9895 to address this issue. best, Colin > > > St.Ack
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
+1 non-binding Verified signatures and checksums one binary and source tar files Built the source Ran some tests on psuedo-distributed cluster. Rob On 08/15/2013 09:15 PM, Arun C Murthy wrote: Folks, I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would like to get released - this fixes the bugs we saw since the last go-around (rc1). The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/ The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
[jira] [Reopened] (MAPREDUCE-5475) MRClientService does not verify ACLs properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reopened MAPREDUCE-5475: --- Reopening, as this appears to be causing problems on a secure cluster. Getting exceptions on the job submission client after the job starts running and it tries to connect to monitor the job: 13/08/22 18:51:59 ERROR security.UserGroupInformation: PriviledgedActionException as:x@y (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.security.AccessControlException: User appattempt_1377189855273_0006_01 cannot perform operation VIEW_JOB Will revert the change for now until we better understand what's going on. > MRClientService does not verify ACLs properly > - > > Key: MAPREDUCE-5475 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5475 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 2.0.4-alpha, 0.23.9 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Fix For: 2.1.1-beta > > Attachments: MAPREDUCE-5475.branch-0.23.patch, MAPREDUCE-5475.patch > > > When MRClientService receives requests, it calls verifyAndGetJob which does > not actually validate that the current user has the proper access. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
RE: [VOTE] Release Apache Hadoop 2.1.0-beta
+1 (non-binding) Bikas -Original Message- From: Hitesh Shah [mailto:hit...@apache.org] Sent: Wednesday, August 21, 2013 4:25 PM To: yarn-...@hadoop.apache.org; Arun Murthy Cc: common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; mapreduce-dev@hadoop.apache.org Subject: Re: [VOTE] Release Apache Hadoop 2.1.0-beta +1 (binding) Verified checksums, built from source and ran basic MR jobs on a single-node cluster. -- Hitesh On Aug 15, 2013, at 2:15 PM, Arun C Murthy wrote: > Folks, > > I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would like to get released - this fixes the bugs we saw since the last go-around (rc1). > > The RC is available at: > http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/ > The RC tag in svn is here: > http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta- > rc2 > > The maven artifacts are available via repository.apache.org. > > Please try the release and vote; the vote will run for the usual 7 days. > > thanks, > Arun > > -- > Arun C. Murthy > Hortonworks Inc. > http://hortonworks.com/ > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or > entity to which it is addressed and may contain information that is > confidential, privileged and exempt from disclosure under applicable > law. If the reader of this message is not the intended recipient, you > are hereby notified that any printing, copying, dissemination, > distribution, disclosure or forwarding of this communication is > strictly prohibited. If you have received this communication in error, > please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [VOTE] Release Apache Hadoop 2.1.0-beta
+1, binding Review process # symlink /usr/local/bin/protoc to the homebrew installed 2.5.0 version # delete all 2.1.0-beta artifacts in the mvn repo: find ~/.m2 -name 2.1.0-beta -print | xargs rm -rf # checkout hbase apache/branch-0.95 (commit # b58d596 ) # switch to ASF repo (arun's private repo is in the POM, with JARs with the same sha1 sum, I'm just being rigorous) ASF Staging https://repository.apache.org/content/groups/staging/ # clean build of hbase tar against the beta artifacts mvn clean install assembly:single -DskipTests -Dmaven.javadoc.skip=true -Dhadoop.profile=2.0 -Dhadoop-two.version=2.1.0-beta # Observe DL taking place ] [INFO] --- maven-assembly-plugin:2.4:single (default-cli) @ hbase --- [INFO] Assemblies have been skipped per configuration of the skipAssembly parameter. [INFO] [INFO] [INFO] Building HBase - Common 0.95.3-SNAPSHOT [INFO] Downloading: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.pom Downloaded: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.pom(2 KB at 3.3 KB/sec) Downloading: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project/2.1.0-beta/hadoop-project-2.1.0-beta.pom Downloaded: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project/2.1.0-beta/hadoop-project-2.1.0-beta.pom(34 KB at 383.8 KB/sec) Downloading: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-main/2.1.0-beta/hadoop-main-2.1.0-beta.pom Downloaded: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-main/2.1.0-beta/hadoop-main-2.1.0-beta.pom(17 KB at 184.0 KB/sec) Downloading: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.pom Downloaded: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.pom(26 KB at 293.3 KB/sec) Downloading: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project-dist/2.1.0-beta/hadoop-project-dist-2.1.0-beta.pom Downloaded: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-project-dist/2.1.0-beta/hadoop-project-dist-2.1.0-beta.pom(17 KB at 176.2 KB/sec) Downloading: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.pom Downloaded: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.pom(7 KB at 79.0 KB/sec) Downloading: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.jar Downloading: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar Downloading: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.jar Downloaded: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-annotations/2.1.0-beta/hadoop-annotations-2.1.0-beta.jar(17 KB at 146.1 KB/sec) Downloaded: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-auth/2.1.0-beta/hadoop-auth-2.1.0-beta.jar(47 KB at 241.7 KB/sec) Downloaded: https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar(2657 KB at 2260.9 KB/sec) [INFO] # get md5 sum of hadoop-common-2.1.0-beta artifact in https://repository.apache.org/content/groups/staging/ : 0166f5c94d3699b3a37efc16ebb1ceea3acb3b53 # verify version of artifact in local m2 repo $ sha1sum ~/.m2/repository/org/apache/hadoop/hadoop-common/2.1.0-beta/hadoop-common-2.1.0-beta.jar 0166f5c94d3699b3a37efc16ebb1ceea3acb3b53 # in hbase/hbase-assembly/target , gunzip then untar the hbase-0.95.3-SNAPSHOT-bin.tar file # Patch the Hoya POM to use 2.1.0-beta instead of a local 2.1.1-SNAPSHOT # run some of the hbase cluster deploy & flexing tests mvn clean test -Pstaging (all tests pass after 20 min) == functional tests = # build and the Hoya JAR with classpath mvn package -Pstaging # D/L the binary .tar.gz file, and scp to an ubuntu VM with the hadoop conf properties for net-accessible HDFS & YARN services, no memory limits on containers https://github.com/hortonworks/hoya/tree/master/src/test/configs/ubuntu https://github.com/hortonworks/hoya/blob/master/src/test/configs/ubuntu/core-site.xml https://github.com/hortonworks/hoya/blob/master/src/test/configs/ubuntu/yarn-site.xml # stop the running hadoop-2.1.1-snapshot cluster # start the new cluster serv
[jira] [Created] (MAPREDUCE-5478) TeraInputFormat unnecessarily defines its own FileSplit subclass
Sandy Ryza created MAPREDUCE-5478: - Summary: TeraInputFormat unnecessarily defines its own FileSplit subclass Key: MAPREDUCE-5478 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5478 Project: Hadoop Map/Reduce Issue Type: Bug Components: examples Affects Versions: 2.1.0-beta Reporter: Sandy Ryza Assignee: Sandy Ryza Priority: Minor TeraInputFormat defines its own TeraFileSplit subclass of FileSplit that adds a locations field, which is already included in FileSplit. This is causing MR2 TeraSort to fail on MR1, which, for a System.arraycopy, requires splits to be of the FileSplit class. While nobody is promising that everything that runs on MR2 should run on MR1, fixing this would be easy and make it possible to compare MR2 TeraSort performance between MR1 and MR2. We should just get rid of TeraFileSplit and use FileSplit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Release Apache Hadoop 2.0.6-alpha (RC1)
I am clearly +1 (non-binding) on the release. With 8 +1s (5 binding), no -1s or 0s the vote passes. Thanks to all who verified the bits, I'll push them out shortly. Thanks, Cos On Thu, Aug 15, 2013 at 10:29PM, Konstantin Boudnik wrote: > All, > > I have created a release candidate (rc1) for hadoop-2.0.6-alpha that I would > like to release. > > This is a stabilization release that includes fixed for a couple a of issues > as outlined on the security list. > > The RC is available at: http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc1/ > The RC tag in svn is here: > http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc1 > > The maven artifacts are available via repository.apache.org. > > The only difference between rc0 and rc1 is ASL added to releasenotes.html and > updated release dates in CHANGES.txt files. > > Please try the release bits and vote; the vote will run for the usual 7 days. > > Thanks for your voting > Cos > signature.asc Description: Digital signature
