Hi Alessandro,
I'll try to explain the details, but I know it is a bit complicated and
not necessarily easy to reproduce.
In my setup I have a master server (4 processors, 16 GB RAM) and 2
slaves (3 processors, 4 GB RAM each) running Hadoop 2.5.1 and ZooKeeper
3.4.6. Thanks to your suggestions I successfully compiled Giraph to use
YARN on top of Hadoop 2.5.1, so I have tried to run the following
command on master:
hadoop jar /home/hadoop/giraph-ex.jar
org.apache.giraph.benchmark.PageRankBenchmark
-Dgiraph.zkList=127.0.0.1:2181 -Dgiraph.pure.yarn.job=true -libjars
/home/hadoop/giraph-core.jar -e 1 -s 1 -v -V 50 -w 1
The job actually starts and, when I visit the job tracking URL, I see
that there is a mapper running. When I click on the mapper, though, and
I look into the logs, I see these lines:
2014-10-28 22:45:38,796 INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: checkWorkers: Only found 0
responses of 1 needed to start superstep -1. Reporting every 30000
msecs, 449879 more msecs left before giving up.
2014-10-28 22:45:38,796 INFO [org.apache.giraph.master.MasterThread]
org.apache.giraph.master.BspServiceMaster: logMissingWorkersOnSuperstep:
No response from partition 1 (could be master)
and on the server the console is stuck here:
14/10/28 22:43:23 INFO mapreduce.Job: Running job: job_1412112271808_0019
14/10/28 22:43:25 INFO mapreduce.Job: Job job_1412112271808_0019 running
in uber mode : false
14/10/28 22:43:25 INFO mapreduce.Job: map 100% reduce 0%
If I change the number of workers to 2 the same happens on one of the
nodes. In the bottom you can see the main configuration files I use. Any
ideas?
Thanks again,
Nicola
------------------------
mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
<description>Execution framework.</description>
</property>
<property>
<name>mapreduce.jobtracker.address</name>
<value>master:8021</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
------------------------
yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
</configuration>
Il 27/10/2014 23:02, Alessandro Negro ha scritto:
> Hi Nicola,
> how are you running the test? Could you help me to test it on yarn?
> I would like to see the error.
>
> Thanks,
> Alessandro
>
> Il giorno 27/ott/2014, alle ore 17:34, Nicola Vitucci
> <[email protected] <mailto:[email protected]>> ha scritto:
>
>> Hi Alessandro,
>>
>> I can now complete the compiling process and get the full .jar. I can't
>> get it working right away, but I think it's more an Hadoop problem than
>> a Giraph problem (can't get past "superstep -1" in the PageRank
>> example). I'll keep you updated.
>>
>> Thanks again,
>>
>> Nicola
>>
>> Il 27/10/2014 17:15, Alessandro Negro ha scritto:
>>> Hi Nicola,
>>> I have just fixed this other issue, but I’m not sure that it will work
>>> completely.
>>> After I checked the code I notice that the issue is related to the munge
>>> symbol STATIC_SASL_SYMBOL.
>>> Try removing STATIC_SASL_SYMBOL from the munge symbols list in the
>>> hadoop_yarn profile.
>>>
>>> My profile was:
>>>
>>> <munge.symbols>PURE_YARN,STATIC_SASL_SYMBOL</munge.symbols>
>>>
>>> now it is:
>>>
>>> <munge.symbols>PURE_YARN</munge.symbols>
>>>
>>>
>>> Now I’m able to continue. As I said I’m not sure that this will produce
>>> a working jar.
>>> I checked into the code and it seems so. If you can please test and let
>>> me know.
>>>
>>> Cheers,
>>> Alessandro
>>>
>>>
>>> Il giorno 27/ott/2014, alle ore 15:52, Nicola Vitucci
>>> <[email protected]
>>> <mailto:[email protected]> <mailto:[email protected]>>
>>> ha scritto:
>>>
>>>> Hi Alessandro,
>>>>
>>>> I can confirm that after your addition I'm able to build Giraph for
>>>> Hadoop 2.4.0, so this works:
>>>>
>>>> mvn package -DskipTests -Dhadoop.version=2.4.0 -Phadoop_yarn clean
>>>> install
>>>>
>>>> Do you know whether there are any plans to support Hadoop 2.5.x too?
>>>>
>>>> Thank you very much,
>>>>
>>>> Nicola
>>>>
>>>> Il 27/10/2014 15:38, Alessandro Negro ha scritto:
>>>>> Hi Nicola,
>>>>> it’s true I got the same error but also before changing the plugin
>>>>> configuration.
>>>>> I was able to compile only for version 2.4.0. I tested with 2.5.0 as
>>>>> well.
>>>>>
>>>>> I tested everything on the main trunk.
>>>>>
>>>>> Cheers,
>>>>> Alessandro
>>>>>
>>>>>
>>>>>
>>>>> Il giorno 27/ott/2014, alle ore 15:00, Nicola Vitucci
>>>>> <[email protected]
>>>>> <mailto:[email protected]> <mailto:[email protected]>> ha
>>>>> scritto:
>>>>>
>>>>>> Hi Alessandro,
>>>>>>
>>>>>> thanks for the reply. In order to start fresh, before adding the line
>>>>>> you suggested I re-cloned the repository and tried to compile from
>>>>>> trunk
>>>>>> using the same command:
>>>>>>
>>>>>> mvn package -DskipTests -Dhadoop.version=2.5.1 -Phadoop_yarn clean
>>>>>> install
>>>>>>
>>>>>> I get an error (on the variable SASL_PROPS) that I could get rid of
>>>>>> before by applying the patch GIRAPH-947, but now the patch is not
>>>>>> applicable any longer, so I can't even get to the stage where the
>>>>>> 'munged' directory was the problem. Can you tell me which branch have
>>>>>> you used and what is the exact sequence of changes that you applied to
>>>>>> get Giraph compiled?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Nicola
>>>>>>
>>>>>> Il 26/10/2014 23:34, Alessandro Negro ha scritto:
>>>>>>> Hi Nicola,
>>>>>>> I was able to fix this issue adding the following line into the
>>>>>>> munge maven plugin:
>>>>>>>
>>>>>>> <mungedDirectory>${project.build.directory}/munged</mungedDirectory>
>>>>>>>
>>>>>>> Now it is:
>>>>>>>
>>>>>>> <plugin>
>>>>>>> <groupId>org.sonatype.plugins</groupId>
>>>>>>> <artifactId>munge-maven-plugin</artifactId>
>>>>>>> <version>1.0</version>
>>>>>>> <executions>
>>>>>>> <execution>
>>>>>>> <id>munge</id>
>>>>>>> <phase>generate-sources</phase>
>>>>>>> <goals>
>>>>>>> <goal>munge</goal>
>>>>>>> </goals>
>>>>>>> <configuration>
>>>>>>>
>>>>>>> <mungedDirectory>${project.build.directory}/munged</mungedDirectory>
>>>>>>> <symbols>${munge.symbols}</symbols>
>>>>>>> </configuration>
>>>>>>> </execution>
>>>>>>> </executions>
>>>>>>> </plugin>
>>>>>>>
>>>>>>> I hope that this will help you.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Alessandro
>
