Re: Propagation of exception message in GraphMapper

2012-03-28 Thread Avery Ching 

Dionysis,

I'm a little confused.  The below message seems to explain the problem with

Input path does not exist: hdfs://localhost:9000/x


Am I missing something?

Avery

On 3/27/12 4:04 AM, Dionysis Logothetis wrote:

Hi all,

I was running one of the example apps and I accidentally entered the
wrong input, so I ended up getting an exception from GraphMapper. But
I think the actual cause of the exception (directory missing) is not
propagated properly, which makes debugging a bit more difficult.
Perhaps the generateInputSplits() method should include the type of
exception in the message?

Below I'm pasting the messages printed:



2012-03-27 12:50:03,925 FATAL org.apache.giraph.graph.GraphMapper:
uncaughtException: OverrideExceptionHandler on thread
org.apache.giraph.graph.MasterThread, msg = generateInputSplits: Got
IOException, exiting...
java.lang.IllegalStateException: generateInputSplits: Got IOException
at 
org.apache.giraph.graph.BspServiceMaster.generateInputSplits(BspServiceMaster.java:249)
at 
org.apache.giraph.graph.BspServiceMaster.createInputSplits(BspServiceMaster.java:506)
at org.apache.giraph.graph.MasterThread.run(MasterThread.java:98)
Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException:
Input path does not exist: hdfs://localhost:9000/x
at 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:231)
at 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:248)
at 
org.apache.giraph.lib.TextVertexInputFormat.getSplits(TextVertexInputFormat.java:120)
at 
org.apache.giraph.graph.BspServiceMaster.generateInputSplits(BspServiceMaster.java:229)
... 2 more
2012-03-27 12:50:03,941 WARN org.apache.giraph.zk.ZooKeeperManager:
onlineZooKeeperServers: Forced a shutdown hook kill of the ZooKeeper
process.

[jira] [Commented] (GIRAPH-169) How to close all child when a job finished?

2012-03-28 Thread Avery Ching (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240558#comment-13240558
 ] 

Avery Ching commented on GIRAPH-169:


That is really strange.  It would be great if could have a very small test case 
(i.e. 2 workers or less) that can consistently reproduce this issue.  Your logs 
look fine.

> How to close all child when a job finished?
> ---
>
> Key: GIRAPH-169
> URL: https://issues.apache.org/jira/browse/GIRAPH-169
> Project: Giraph
>  Issue Type: Improvement
>  Components: mapreduce
>Affects Versions: 0.2.0
> Environment: sles 11 x64,jdk 1.6,hadoop 0.20.205.0,1 Master and 8 
> slaves,
>Reporter: Jianfeng Qian
>Priority: Minor
>
> I ran pagerank at hadoop 0.20.205.0. When the job finished,the child in 
> slaves didn't quit immediately and sometimes they never quit and I have to 
> kill them. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (GIRAPH-169) How to close all child when a job finished?

2012-03-28 Thread Jianfeng Qian (Commented) (JIRA) 

[ 
https://issues.apache.org/jira/browse/GIRAPH-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240285#comment-13240285
 ] 

Jianfeng Qian commented on GIRAPH-169:
--

hadoop 0.20.205.0. most of the time, the worker can't quit.
sorry, there is the full worker log
2012-03-28 10:18:00,122 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to 
load native-hadoop library for your platform... using builtin-java classes 
where applicable
2012-03-28 10:18:00,387 WARN org.apache.giraph.bsp.BspOutputFormat: 
getOutputCommitter: Returning ImmutableOutputCommiter (does nothing).
2012-03-28 10:18:00,397 INFO org.apache.hadoop.util.ProcessTree: setsid exited 
with exit code 0
2012-03-28 10:18:00,405 INFO org.apache.hadoop.mapred.Task:  Using 
ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@18330bf
2012-03-28 10:18:00,489 INFO org.apache.giraph.graph.GraphMapper: Distributed 
cache is empty. Assuming fatjar.
2012-03-28 10:18:00,489 INFO org.apache.giraph.graph.GraphMapper: setup: 
classpath @ 
/usr/local/test-0302/hadoop-data/h-0.20.205/mapred/local/taskTracker/root/jobcache/job_201203281017_0001/jars/job.jar
2012-03-28 10:18:00,498 INFO org.apache.giraph.zk.ZooKeeperManager: 
createCandidateStamp: Made the directory 
_bsp/_defaultZkManagerDir/job_201203281017_0001
2012-03-28 10:18:00,500 INFO org.apache.giraph.zk.ZooKeeperManager: 
createCandidateStamp: Creating my filestamp 
_bsp/_defaultZkManagerDir/job_201203281017_0001/_task/tmm-e6 1
2012-03-28 10:18:00,521 INFO org.apache.giraph.zk.ZooKeeperManager: 
getZooKeeperServerList: For task 1, got file 'zkServerList_tmm-e10 0 ' (polling 
period is 3000)
2012-03-28 10:18:00,521 INFO org.apache.giraph.zk.ZooKeeperManager: 
getZooKeeperServerList: Found [tmm-e10, 0] 2 hosts in filename 
'zkServerList_tmm-e10 0 '
2012-03-28 10:18:00,524 INFO org.apache.giraph.zk.ZooKeeperManager: 
onlineZooKeeperServers: Got [tmm-e10] 1 hosts from 1 ready servers when 1 
required (polling period is 3000) on attempt 0
2012-03-28 10:18:00,524 INFO org.apache.giraph.graph.GraphMapper: setup: 
Starting up BspServiceWorker...
2012-03-28 10:18:00,534 INFO org.apache.giraph.graph.BspService: BspService: 
Connecting to ZooKeeper with job job_201203281017_0001, 1 on tmm-e10:22181
2012-03-28 10:18:00,540 INFO org.apache.zookeeper.ZooKeeper: Client 
environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
2012-03-28 10:18:00,540 INFO org.apache.zookeeper.ZooKeeper: Client 
environment:host.name=tmm-e6
2012-03-28 10:18:00,540 INFO org.apache.zookeeper.ZooKeeper: Client 
environment:java.version=1.6.0_22
2012-03-28 10:18:00,540 INFO org.apache.zookeeper.ZooKeeper: Client 
environment:java.vendor=Sun Microsystems Inc.
2012-03-28 10:18:00,540 INFO org.apache.zookeeper.ZooKeeper: Client 
environment:java.home=/usr/local/java/jdk1.6.0_22/jre
2012-03-28 10:18:00,540 INFO org.apache.zookeeper.ZooKeeper: Client 
environment:java.class.path=/usr/local/test-0302/hadoop-data/h-0.20.205/mapred/local/taskTracker/root/jobcache/job_201203281017_0001/jars/classes:/usr/local/test-0302/hadoop-data/h-0.20.205/mapred/local/taskTracker/root/jobcache/job_201203281017_0001/jars:/usr/local/test-0302/hadoop-data/h-0.20.205/mapred/local/taskTracker/root/jobcache/job_201203281017_0001/attempt_201203281017_0001_m_01_0/work:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../conf:/usr/local/java/jdk1.6.0_22/lib/tools.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/hadoop-core-0.20.205.0.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/asm-3.2.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/aspectjrt-1.6.5.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/aspectjtools-1.6.5.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/commons-cli-1.2.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/commons-codec-1.4.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/commons-collections-3.2.1.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/commons-configuration-1.6.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/commons-daemon-1.0.1.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/commons-digester-1.8.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/commons-el-1.0.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/commons-httpclient-3.0.1.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/commons-lang-2.4.jar:/usr/local/test-0302/hadoop-0.20.205.0/libexec/../share/hadoop/lib/commons