[
https://issues.apache.org/jira/browse/GIRAPH-302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13436909#comment-13436909
]
Avery Ching commented on GIRAPH-302:
------------------------------------
Thanks for the quick review Alessandro!
We can trigger a manual rebuild by logging in to hudson (upper right corner)
and then clicking "build now" (which I just did). The error here was a strange
ZK reconnect (we have this periodically). Rebuilding should fix it.
12/08/17 10:18:30 INFO zookeeper.ClientCnxn: Unable to read additional data
from server sessionid 0x139341759650003, likely server has closed socket,
closing socket connection and attempting reconnect
12/08/17 10:18:30 WARN zookeeper.ClientCnxn: Session 0x139341759650002 for
server janus.apache.org/67.195.138.60:22181, unexpected error, closing socket
connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:200)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
12/08/17 10:18:30 ERROR zookeeper.ClientCnxn: Error while calling watcher
java.lang.RuntimeException: process: Disconnected from ZooKeeper, cannot
recover - WatchedEvent state:Disconnected type:None path:null
at org.apache.giraph.graph.BspService.process(BspService.java:1010)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
12/08/17 10:18:30 FATAL graph.GraphMapper: uncaughtException:
OverrideExceptionHandler on thread org.apache.giraph.graph.MasterThread, msg =
barrierOnWorkerList: KeeperException - Couldn't get children of
/_hadoopBsp/job_local_0008/_applicationAttemptsDir/0/_superstepDir/5/_workerFinishedDir,
exiting...
java.lang.IllegalStateException: barrierOnWorkerList: KeeperException -
Couldn't get children of
/_hadoopBsp/job_local_0008/_applicationAttemptsDir/0/_superstepDir/5/_workerFinishedDir
at
org.apache.giraph.graph.BspServiceMaster.barrierOnWorkerList(BspServiceMaster.java:1317)
at
org.apache.giraph.graph.BspServiceMaster.coordinateSuperstep(BspServiceMaster.java:1507)
at org.apache.giraph.graph.MasterThread.run(MasterThread.java:111)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for
/_hadoopBsp/job_local_0008/_applicationAttemptsDir/0/_superstepDir/5/_workerFinishedDir
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1243)
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1271)
at
org.apache.giraph.zk.ZooKeeperExt.getChildrenExt(ZooKeeperExt.java:273)
at
org.apache.giraph.graph.BspServiceMaster.barrierOnWorkerList(BspServiceMaster.java:1311)
... 2 more
> Thread safety issue with sending partitions around
> --------------------------------------------------
>
> Key: GIRAPH-302
> URL: https://issues.apache.org/jira/browse/GIRAPH-302
> Project: Giraph
> Issue Type: Bug
> Reporter: Avery Ching
> Assignee: Avery Ching
> Priority: Critical
> Attachments: GIRAPH-302.patch
>
>
> When calling sendPartitionRequest(), we clear the vertex list afterward,
> making it a race!
> I noticed this when I was running with 300 workers and the number of edges
> wasn't what I expected. Sometimes we get empty requests!
> After digging into the code I found the issue and have fixed it.
> Giraph Stats Aggregate edges 99,971,220 0 99,971,220
> Superstep 11 0 11
> Current workers 300 0 300
> Last checkpointed superstep 0 0 0
> Current master task partition 0 0 0
> Sent messages 0 0 0
> Aggregate finished vertices 10,000,000 0 10,000,000
> Aggregate vertices 10,000,000 0 10,000,000
> This is wrong!
> Giraph Stats Aggregate edges 100,000,000 0 100,000,000
> Superstep 11 0 11
> Last checkpointed superstep 0 0 0
> Current workers 300 0 300
> Current master task partition 0 0 0
> Sent messages 0 0 0
> Aggregate finished vertices 10,000,000 0 10,000,000
> Aggregate vertices 10,000,000 0 10,000,000
> Fixed!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
