Thanks. Recompiling common and hdfs by commands
1st. compile commons
ant veryclean mvn-install (hadoop-common-0.22.0-SNAPSHOT.jar is
installed to m2 repository)
2nd. compile hdfs
ant veryclean mvn-install -Dresolvers=internal (It installs
hadoop-hdfs-0.22.0-SNAPSHOT.jar to repository)
seem to solve the problem below
compile-hdfs-classes:
[javac] mr-279/hdfs/build.xml:339: warning: 'includeantruntime' was
not set, defaulting to build.sysclasspath=last; set to false for
repeatable builds
[javac] Compiling 237 source files to mr-279/hdfs/build/classes
[javac]
mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:119:
cannot find symbol
...
Now the issue seems to be how to build avro-maven-plugin-1.4.0?
I execute the command
export MAVEN_OPTS=-Xmx512m
mvn clean install assembly:assembly
would produce error
[ERROR] Failed to execute goal
org.codehaus.mojo:exec-maven-plugin:1.2:exec (generate-sources) on
project yarn-api: Command execution failed. Process exited with an
error: 127(Exit value: 127) -> [Help 1]
But the avro maven plugin obtained from
https://github.com/phunt/avro-maven-plugin.git only provide 1.0. Would
modify version (pointing it to 1.4.0) in pom work? Or what is the next
step for building yarn-api?
Thanks for help.
On Fri, Jun 17, 2011 at 9:09 PM, Thomas Graves <[email protected]> wrote:
> Did you build common and hdfs before doing mvn install in mapreduce? You
> have to build them in order as stated in INSTALL doc - common, hdfs, then
> mapreduce.
>
> Tom
>
>
> On 6/17/11 3:33 AM, "Thomas Anderson" <[email protected]> wrote:
>
>> I was not aware that the source downloaded (a few months ago) is
>> obsoleted. So now I switch by doing svn update, which solves the code
>> obsolete issue.
>>
>> Then I follow the instruction at mapreduce/INSTALL [1], which points
>> to install depdencies for yarn first at README[2]. During searching
>> mailing list, it seems avro plugin is not necessary to install
>> manually. So I only install protobuf 2.4.1 (configure/ make/ make
>> install works ok.) But after that, a mvn install under mapreduce
>> produces error
>>
>> Failed to execute goal on project yarn-api: Could not resolve
>> dependencies for project org.apache.hadoop:yarn-api:jar:1.0-SNAPSHOT:
>> Failure to find org.apache.hadoop:hadoop-hdfs:jar:0.22.0-SNAPSHOT in
>> https://repository.jboss.org/nexus/content/groups/public-jboss/ was
>> cached in the local repository, resolution will not be reattempted
>> until the update interval of jboss-public-repository-group has elapsed
>> or updates are forced -> [Help 1]
>>
>> That looks like missing the hdfs jar artifact. So cd to mr-279/hdfs
>> and execute `ant clean package` generates error message as shown in
>> compile-hdfs-classes section.
>>
>> What is the right order/ procedure to successfully compile mr-279? Or
>> will have a new update for the instruction?
>>
>> Thanks for help.
>>
>> [1]. mapreduce/INSTALL.
>>
> http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTAL>
> L
>> [2]. README.
>> http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/yarn/R
>> EADME
>>
>>
>> compile-hdfs-classes:
>> [javac] mr-279/hdfs/build.xml:339: warning: 'includeantruntime'
>> was not set, defaulting to build.sysclasspath=last; set to false for
>> repeatable builds
>> [javac] Compiling 237 source files to mr-279/hdfs/build/classes
>> [javac]
>> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:119:
>> cannot find symbol
>> [javac] symbol : class ProtocolSignature
>> [javac] location: package org.apache.hadoop.ipc
>> [javac] import org.apache.hadoop.ipc.ProtocolSignature;
>> [javac] ^
>> [javac]
>>
> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:2308>
> :
>> cannot find symbol
>> [javac] symbol : class ProtocolSignature
>> [javac] location: class org.apache.hadoop.hdfs.server.datanode.DataNode
>> [javac] public ProtocolSignature getProtocolSignature(String protocol,
>> [javac] ^
>> [javac]
>> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:85:
>> cannot find symbol
>> [javac] symbol : class ProtocolSignature
>> [javac] location: package org.apache.hadoop.ipc
>> [javac] import org.apache.hadoop.ipc.ProtocolSignature;
>> [javac] ^
>> [javac]
>> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:194:
>> cannot find symbol
>> [javac] symbol : class ProtocolSignature
>> [javac] location: class org.apache.hadoop.hdfs.server.namenode.NameNode
>> [javac] public ProtocolSignature getProtocolSignature(String protocol,
>> [javac] ^
>> [javac] mr-279/hdfs/src/java/org/apache/hadoop/fs/Hdfs.java:389:
>> cannot find symbol
>> [javac] symbol : method getCanonicalServiceName()
>> [javac] location: class org.apache.hadoop.fs.Hdfs
>> [javac] result.setService(new Text(this.getCanonicalServiceName()));
>> [javac] ^
>> [javac] mr-279/hdfs/src/java/org/apache/hadoop/fs/Hdfs.java:385:
>> method does not override or implement a method from a supertype
>> [javac] @Override //AbstractFileSystem
>> [javac] ^
>> [javac] mr-279/hdfs/src/java/org/apache/hadoop/hdfs/DFSClient.java:736:
>> cannot find symbol
>> [javac] symbol : method
>> validate(java.util.EnumSet<org.apache.hadoop.fs.CreateFlag>)
>> [javac] location: class org.apache.hadoop.fs.CreateFlag
>> [javac] CreateFlag.validate(flag);
>> [javac] ^
>> [javac]
>> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
>> 5210:
>> cannot find symbol
>> [javac] symbol : method isRpcInvocation()
>> [javac] location: class org.apache.hadoop.ipc.Server
>> [javac] return Server.isRpcInvocation();
>> [javac] ^
>> [javac]
>>
> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java:102>
> :
>> cannot find symbol
>> [javac] symbol : method listFiles(java.io.File)
>> [javac] location: class org.apache.hadoop.fs.FileUtil
>> [javac] File[] files = FileUtil.listFiles(dir);
>> [javac] ^
>> [javac]
>>
> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java:190>
> :
>> cannot find symbol
>> [javac] symbol : method listFiles(java.io.File)
>> [javac] location: class org.apache.hadoop.fs.FileUtil
>> [javac] File files[] = FileUtil.listFiles(dir);
>> [javac] ^
>> [javac]
>>
> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java:425>
> :
>> cannot find symbol
>> [javac] symbol : method listFiles(java.io.File)
>> [javac] location: class org.apache.hadoop.fs.FileUtil
>> [javac] File blockFiles[] = FileUtil.listFiles(dir);
>> [javac] ^
>> [javac]
>>
> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java:726>
> :
>> cannot find symbol
>> [javac] symbol : method list(java.io.File)
>> [javac] location: class org.apache.hadoop.fs.FileUtil
>> [javac] if (finalizedDir.exists() &&
>> FileUtil.list(finalizedDir).length != 0) {
>> [javac] ^
>> [javac]
>>
> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java:729>
> :
>> cannot find symbol
>> [javac] symbol : method list(java.io.File)
>> [javac] location: class org.apache.hadoop.fs.FileUtil
>> [javac] if (rbwDir.exists() && FileUtil.list(rbwDir).length != 0) {
>> [javac] ^
>> [javac]
>>
> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java:758>
> :
>> cannot find symbol
>> [javac] symbol : method listFiles(java.io.File)
>> [javac] location: class org.apache.hadoop.fs.FileUtil
>> [javac] for (File f : FileUtil.listFiles(bpCurrentDir)) {
>> [javac] ^
>> [javac]
>>
> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java:766>
> :
>> cannot find symbol
>> [javac] symbol : method listFiles(java.io.File)
>> [javac] location: class org.apache.hadoop.fs.FileUtil
>> [javac] for (File f : FileUtil.listFiles(bpDir)) {
>> [javac] ^
>> [javac]
>>
> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:2310>
> :
>> cannot find symbol
>> [javac] symbol : variable ProtocolSignature
>> [javac] location: class org.apache.hadoop.hdfs.server.datanode.DataNode
>> [javac] return ProtocolSignature.getProtocolSigature(
>> [javac] ^
>> [javac]
>>
> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:2307>
> :
>> method does not override or implement a method from a supertype
>> [javac] @Override
>> [javac] ^
>> [javac]
>> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java:5
>> 05:
>> cannot find symbol
>> [javac] symbol : method list(java.io.File)
>> [javac] location: class org.apache.hadoop.fs.FileUtil
>> [javac] if (FileUtil.list(detachDir).length != 0 ) {
>> [javac] ^
>> [javac]
>> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.j
>> ava:490:
>> cannot find symbol
>> [javac] symbol : method listFiles(java.io.File)
>> [javac] location: class org.apache.hadoop.fs.FileUtil
>> [javac] files = FileUtil.listFiles(dir);
>> [javac] ^
>> [javac]
>> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStor
>> age.java:355:
>> cannot find symbol
>> [javac] symbol : method list(java.io.File)
>> [javac] location: class org.apache.hadoop.fs.FileUtil
>> [javac] if (FileUtil.list(detachDir).length != 0) {
>> [javac] ^
>> [javac]
>> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:812:
>> method does not override or implement a method from a supertype
>> [javac] @Override // FileSystem
>> [javac] ^
>> [javac]
>> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:196:
>> cannot find symbol
>> [javac] symbol : variable ProtocolSignature
>> [javac] location: class org.apache.hadoop.hdfs.server.namenode.NameNode
>> [javac] return ProtocolSignature.getProtocolSigature(
>> [javac] ^
>> [javac]
>> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:193:
>> method does not override or implement a method from a supertype
>> [javac] @Override
>> [javac] ^
>> [javac]
>> mr-279/hdfs/src/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java:273:
>> cannot find symbol
>> [javac] symbol : method getFS()
>> [javac] location: class org.apache.hadoop.hdfs.tools.DFSAdmin
>> [javac] FileSystem fs = getFS();
>> [javac] ^
>> [javac] Note: Some input files use or override a deprecated API.
>> [javac] Note: Recompile with -Xlint:deprecation for details.
>> [javac] 24 errors
>>
>>
>>
>>
>> On Thu, Jun 16, 2011 at 4:33 PM, Luke Lu <[email protected]> wrote:
>>> Why are you compiling branch HDFS-1052?, it was a temporary branch for
>>> merging federated NN changes into trunk. All the changes in that
>>> branch have since been merged into trunk and MR-279. These stale
>>> branches should be deleted, IMO.
>>>
>>> MR-279 only supports the common and hdfs in the same branch, though
>>> we're moving to trunk soon.
>>>
>>> On Thu, Jun 16, 2011 at 12:04 AM, Thomas Anderson
>>> <[email protected]> wrote:
>>>> When following mr-279/INSTALL to compile source, it throws the
>>>> following error in the second step to compile HDFS-1052. How to solve
>>>> this problem?
>>>> ...
>>>
>
>
