Executing this doesn't seem to work either. C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell>hadoop jar target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar org.apache.hadoop.yarn.applications.distributedshell.Client -jar target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar -shell_command whoami
Thank you for pointing out the classpath issue, though. I checked in hadoop-dist/.../share and found that hadoop had been using the jar there. The command can't find the Client class unless it's in that directory, so I suppose a suitable workaround would be to add the distributedshell/target directory to the classpath myself, or just copy the jar to /share. If anyone has suggestions for the best way to get around that issue, I would welcome them. Otherwise, this relieves my confusion--thanks for all the help! Curtis On Fri, Jun 21, 2013 at 5:10 PM, Hitesh Shah <[email protected]> wrote: > Hello Curtis > > Try the following: > > hadoop jar ./target/distributed-shell.jar.rebuilt-one > org.apache.hadoop.yarn.applications.distributedshell.Client -jar ... > > If you are running hadoop without the jar command, it will find the first > instance of Client.class in its classpath which I am guessing is from the > jars used to start the cluster and not the re-built jar. > > -- Hitesh > > > On Jun 21, 2013, at 4:09 PM, Curtis Ullerich wrote: > > > I should clarify that the build doesn't actually fail (so that bug > doesn't seem to be the issue). It reports success. But then the output from > the just-build jar doesn't change. > > > > Here's my output from building: > > > > > C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell>mvn > clean package -DskipTests > > [INFO] Scanning for projects... > > [INFO] > > [INFO] > ------------------------------------------------------------------------ > > [INFO] Building hadoop-yarn-applications-distributedshell 3.0.0-SNAPSHOT > > [INFO] > ------------------------------------------------------------------------ > > [INFO] > > [INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ > hadoop-yarn-applications-distributedshell --- > > [INFO] Deleting > C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target > > [INFO] > > [INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ > hadoop-yarn-applications-distributedshell --- > > [INFO] Executing tasks > > > > main: > > [mkdir] Created dir: > C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target\test-dir > > [INFO] Executed tasks > > [INFO] > > [INFO] --- maven-resources-plugin:2.2:resources (default-resources) @ > hadoop-yarn-applications-distributedshell --- > > [INFO] Using default encoding to copy filtered resources. > > [INFO] > > [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ > hadoop-yarn-applications-distributedshell --- > > [INFO] Compiling 3 source files to > C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target\classes > > [INFO] > > [INFO] --- maven-resources-plugin:2.2:testResources > (default-testResources) @ hadoop-yarn-applications-distributedshell --- > > [INFO] Using default encoding to copy filtered resources. > > [INFO] > > [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) > @ hadoop-yarn-applications-distributedshell --- > > [INFO] Compiling 1 source file to > C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target\test-classes > > [INFO] > > [INFO] --- maven-jar-plugin:2.3.1:jar (default) @ > hadoop-yarn-applications-distributedshell --- > > [INFO] Building jar: > C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell\target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar > > [INFO] > > [INFO] --- maven-surefire-plugin:2.12.3:test (default-test) @ > hadoop-yarn-applications-distributedshell --- > > [INFO] Tests are skipped. > > [INFO] > > [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ > hadoop-yarn-applications-distributedshell --- > > [INFO] > > [INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ > hadoop-yarn-applications-distributedshell --- > > [INFO] > ------------------------------------------------------------------------ > > [INFO] BUILD SUCCESS > > [INFO] > ------------------------------------------------------------------------ > > [INFO] Total time: 3.487s > > [INFO] Finished at: Fri Jun 21 16:08:02 PDT 2013 > > [INFO] Final Memory: 19M/367M > > [INFO] > ------------------------------------------------------------------------ > > > > > > > > On Fri, Jun 21, 2013 at 3:57 PM, Arpit Agarwal <[email protected]> > wrote: > > Curtis, if you are running 'mvn clean' in a subproject you may be > hitting this bug: > > https://issues.apache.org/jira/browse/HADOOP-9383 > > > > You might have better luck if you include install along with package in > the same invocation. > > > > -Arpit > > > > > > On Fri, Jun 21, 2013 at 3:12 PM, Curtis Ullerich <[email protected]> > wrote: > > I've executed the commands as you've said, and the jar that is run is > unchanged. Here's exactly what I did. > > > > C:\l\> start-dfs > > C:\l\> start-yarn > > C:\l\> cd > hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell > > > C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell> > hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar > target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar > -shell_command whoami > > ...output... > > ...modify logging code in > C:/l/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java > > > C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell> > mvn clean install -DskipTests > > > C:\l\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell> > hadoop org.apache.hadoop.yarn.applications.distributedshell.Client -jar > target\hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar > -shell_command whoami > > ...identical output to before... > > > > Do you see anything wrong with what I've done? > > > > Thanks, > > Curtis > > > > > > > > > > On Thu, Jun 20, 2013 at 7:17 PM, Omkar Joshi <[email protected]> > wrote: > > Hi Curtis, > > > > where are you picking your jar file from? once you run above command you > will see the updated jar file in > > > "..../hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/target/hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar" > > > > I hope you are not using below jar file > > > "..../hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.0.0-SNAPSHOT.jar" > > > > > > What chris has said is right. > > > > once you have taken latest code, you should follow below steps > > 1) mvn clean install -DskipTests (clean to remove previously generated > code) > > 2) now say you are updating distributed shell client code. then go to > ""..../hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/" > and then run "mvn clean install" and use jar from target sub folder. > > > > Thanks, > > Omkar Joshi > > Hortonworks Inc. > > > > > > On Thu, Jun 20, 2013 at 11:47 AM, Curtis Ullerich < > [email protected]> wrote: > > Hi Chris, > > > > I really appreciate the response. What you described is what I initially > tried. The changes never seem to take effect though. Here's what I've done > (this is Windows): > > > > cd %hadoop_install_dir% > > mvn clean package -DskipTests > > mvn install -DskipTests > > > > --- modify the code in distributed shell's Client.java --- > > > > cd > hadoop-yarn-project\hadoop-yarn\hadoop-yarn-applications\hadoop-yarn-applications-distributedshell > > mvn clean > > mvn package -DskipTests > > mvn install -DskipTests > > > > Then I run the jar, just as before. I've just been changing log > statements to see if the changes worked. They don't--the output doesn't > change after doing this rebuild. I've also tried clearing the files put on > HDFS in my user directory in case that was the issue. I've taken a more > thorough look at BUILDING.txt and I seemed to be consistent with the > procedures described there. Am I missing anything else? I've tried > restarting yarn and dfs, though I didn't think that would matter. > > > > Thanks, > > Curtis > > > > > > > > > > On Thu, Jun 20, 2013 at 11:17 AM, Chris Nauroth < > [email protected]> wrote: > > Hi Curtis, > > > > I handle this by running "mvn install -DskipTests" once from the root of > the whole hadoop project to install the sub-modules in my local Maven > repository. Then, you can change your working directory to any sub-module > (like distributed shell) and run a successful Maven build for just that > sub-module. Full details are in the BUILDING.txt file in the root of the > hadoop project. Look for the section titled "Building components > separately". I hope this helps. > > > > Chris Nauroth > > Hortonworks > > http://hortonworks.com/ > > > > > > > > On Wed, Jun 19, 2013 at 4:53 PM, Curtis Ullerich <[email protected]> > wrote: > > What is the way to compile just a subproject of Hadoop with Maven > (without rebuilding parent modules)? In my case I'm wondering about > DistributedShell. I thought that this would have worked but no changes seem > to take effect after building this way. > > > > Thanks, > > Curtis > > > > > > > > > > > > > >
