Great! Absolutely a better approach. Thanks
At 2015-12-23 01:49:08, "Chris Nauroth" <[email protected]> wrote: >If you want the capability to run live pseudo-distributed and deploy code >changes without doing a full distro tarball build, then you can control >the classpath by setting a few more environment variables in >hadoop-env.sh. Here is an example of what I'm doing in one of my dev >environments. > >export HADOOP_USER_CLASSPATH_FIRST=1 >HADOOP_REPO=~/git/hadoop >export >HADOOP_CLASSPATH=$HADOOP_REPO/hadoop-common-project/hadoop-common/target/cl >asses:$HADOOP_REPO/hadoop-hdfs-project/hadoop-hdfs-client/target/classes:$H >ADOOP_REPO/hadoop-hdfs-project/hadoop-hdfs/target/classes > > >Setting HADOOP_CLASSPATH adds additional paths to the classpath before the >shell launches the JVM. In my case, I have the source checked out to >~/git/hadoop, and I point to the target/classes sub-directories for the >individual sub-modules that I want to override and test. Then, I can make >code changes, run "mvn compile" in the sub-module directory, and restart >the daemons. > >By default, the HADOOP_CLASSPATH entries are added at the end of the >standard classpath. Setting HADOOP_USER_CLASSPATH_FIRST=1 changes that >behavior so that the custom entries are first. This way, my built code >changes override the classes that were bundled in the tarball distro. > >--Chris Nauroth > > > > >On 12/21/15, 7:29 PM, "Allen Zhang" <[email protected]> wrote: > >> >>oh, so cool. awesome. Thanks >> >> >> >> >> >> >> >>At 2015-12-22 11:01:55, "Jeff Zhang" <[email protected]> wrote: >>>If you want to change the yarn internal code, you can use MiniYarnCluster >>>for testing. >>> >>>https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-ya >>>rn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/h >>>adoop/yarn/server/MiniYARNCluster.java >>> >>>On Tue, Dec 22, 2015 at 10:00 AM, Allen Zhang <[email protected]> >>>wrote: >>> >>>> >>>> >>>> so, does it to mean that, if I change or add some code, I have to >>>> re-tarball the whole project using "mvn clean package -Pdist >>>>-DskipTests >>>> -Dtar", and then, deploy it to somewhere to remote debug? if yes, I >>>>think >>>> it is so inconvincence. if no, can you guys explain more in this way? >>>> >>>> >>>> Thanks, >>>> Allen >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> At 2015-12-22 08:55:01, "Jeff Zhang" <[email protected]> wrote: >>>> >+1 for Chris, remote debug will help you. >>>> > >>>> >On Tue, Dec 22, 2015 at 1:54 AM, Chris Nauroth >>>><[email protected]> >>>> >wrote: >>>> > >>>> >> If you're running the Hadoop daemons in pseudo-distributed mode >>>>(all the >>>> >> daemons running as separate processes, but on a single dev host), >>>>then >>>> >> another option is to launch the daemon's JVM with the JDWP >>>>arguments and >>>> >> attach a "remote" debugger. This can be either the jdb CLI debugger >>>> that >>>> >> ships with the JDK or a fancier IDE like Eclipse or IntelliJ. >>>> >> >>>> >> Each daemon's JVM arguments are controlled with an environment >>>>variable >>>> >> suffixed with "_OPTS" defined in files named *-env.sh. For >>>>example, in >>>> >> hadoop-env.sh, you could set something like this to enable remote >>>> >> debugging for the NameNode process: >>>> >> >>>> >> export >>>> >> >>>> >>>>HADOOP_NAMENODE_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,addres >>>>s=8 >>>> >> 000,suspend=n $HADOOP_NAMENODE_OPTS" >>>> >> >>>> >> >>>> >> Then, you can run "jdb -attach localhost:8000" to attach the >>>>debugger, >>>> or >>>> >> do the equivalent in your IDE of choice. >>>> >> >>>> >> --Chris Nauroth >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> On 12/21/15, 7:25 AM, "Daniel Templeton" <[email protected]> >>>>wrote: >>>> >> >>>> >> >Your best bet is to find a test that includes all the bits you >>>>want and >>>> >> >execute that test in debug mode. (You can also change an existing >>>>test >>>> >> >to include what you want, but in most cases it is easier to start >>>>with >>>> >> >an existing test than to start from scratch.) >>>> >> > >>>> >> >Daniel >>>> >> > >>>> >> >On 12/20/15 6:01 PM, Allen Zhang wrote: >>>> >> >> Hi all, >>>> >> >> >>>> >> >> I am reading hadoop-2.6.0 source code, mainly focusing on hadoop >>>> yarn. >>>> >> >> However i have some problems in reading or debugging the source >>>> >> >>code,can I debug it locally(I mean in my laptop locally with this >>>> source >>>> >> >>code I've downloaded, not remotely debug), >>>> >> >> because I need to track it execution flow stey by stey, and then >>>>I >>>> want >>>> >> >>to add a new feature or enhancement. >>>> >> >> >>>> >> >> >>>> >> >> So can anyone give some good suggestions or share your method or >>>>any >>>> >> >>wiki page? Really appreciate!! >>>> >> >> >>>> >> >> >>>> >> >> Thanks, >>>> >> >> Allen >>>> >> > >>>> >> > >>>> >> >>>> >> >>>> > >>>> > >>>> >-- >>>> >Best Regards >>>> > >>>> >Jeff Zhang >>>> >>> >>> >>> >>>-- >>>Best Regards >>> >>>Jeff Zhang >
