idealboy commented on issue #7396: **!!!Error with dist-sync on two
machines, Thank you
URL:
https://github.com/apache/incubator-mxnet/issues/7396#issuecomment-321313876
This is the output when "export PS_VERBOSE=2"
[00:38:29] src/van.cc:161: ? => 1. Meta: request=0, timestamp=0, control={
cmd=ADD_NODE, node={ role=server, ip=10.15.240.189, port=52099, is_recovery=0 }
}
[00:38:29] src/van.cc:161: ? => 1. Meta: request=0, timestamp=0, control={
cmd=ADD_NODE, node={ role=worker, ip=10.15.240.189, port=39902, is_recovery=0 }
}
[00:38:29] src/van.cc:161: ? => 1. Meta: request=0, timestamp=0, control={
cmd=ADD_NODE, node={ role=server, ip=10.15.133.82, port=35715, is_recovery=0 } }
[00:38:29] src/van.cc:161: ? => 1. Meta: request=0, timestamp=0, control={
cmd=ADD_NODE, node={ role=worker, ip=10.15.133.82, port=37993, is_recovery=0 } }
[00:38:29] src/van.cc:235: assign rank=9 to node role=worker,
ip=10.15.240.189, port=39902, is_recovery=0
[00:38:29] src/van.cc:235: assign rank=8 to node role=server,
ip=10.15.240.189, port=52099, is_recovery=0
[00:38:29] src/van.cc:235: assign rank=10 to node role=server,
ip=10.15.133.82, port=35715, is_recovery=0
[00:38:29] src/van.cc:235: assign rank=11 to node role=worker,
ip=10.15.133.82, port=37993, is_recovery=0
[00:38:29] src/van.cc:136: ? => 9. Meta: request=0, timestamp=0, control={
cmd=ADD_NODE, node={ role=worker, id=9, ip=10.15.240.189, port=39902,
is_recovery=0 role=server, id=8, ip=10.15.240.189, port=52099, is_recovery=0
role=server, id=10, ip=10.15.133.82, port=35715, is_recovery=0 role=worker,
id=11, ip=10.15.133.82, port=37993, is_recovery=0 role=scheduler, id=1,
ip=10.15.240.189, port=9118, is_recovery=0 } }
[00:38:29] src/van.cc:136: ? => 11. Meta: request=0, timestamp=1, control={
cmd=ADD_NODE, node={ role=worker, id=9, ip=10.15.240.189, port=39902,
is_recovery=0 role=server, id=8, ip=10.15.240.189, port=52099, is_recovery=0
role=server, id=10, ip=10.15.133.82, port=35715, is_recovery=0 role=worker,
id=11, ip=10.15.133.82, port=37993, is_recovery=0 role=scheduler, id=1,
ip=10.15.240.189, port=9118, is_recovery=0 } }
[00:38:29] src/van.cc:136: ? => 8. Meta: request=0, timestamp=2, control={
cmd=ADD_NODE, node={ role=worker, id=9, ip=10.15.240.189, port=39902,
is_recovery=0 role=server, id=8, ip=10.15.240.189, port=52099, is_recovery=0
role=server, id=10, ip=10.15.133.82, port=35715, is_recovery=0 role=worker,
id=11, ip=10.15.133.82, port=37993, is_recovery=0 role=scheduler, id=1,
ip=10.15.240.189, port=9118, is_recovery=0 } }
[00:38:29] src/van.cc:136: ? => 10. Meta: request=0, timestamp=3, control={
cmd=ADD_NODE, node={ role=worker, id=9, ip=10.15.240.189, port=39902,
is_recovery=0 role=server, id=8, ip=10.15.240.189, port=52099, is_recovery=0
role=server, id=10, ip=10.15.133.82, port=35715, is_recovery=0 role=worker,
id=11, ip=10.15.133.82, port=37993, is_recovery=0 role=scheduler, id=1,
ip=10.15.240.189, port=9118, is_recovery=0 } }
[00:38:29] src/van.cc:251: the scheduler is connected to 2 workers and 2
servers
[00:38:29] src/van.cc:161: 9 => 1. Meta: request=1, timestamp=1, control={
cmd=BARRIER, barrier_group=7 }
[00:38:29] src/van.cc:291: Barrier count for 7 : 1
[00:38:29] src/van.cc:161: 8 => 1. Meta: request=1, timestamp=1, control={
cmd=BARRIER, barrier_group=7 }
[00:38:29] src/van.cc:291: Barrier count for 7 : 2
[00:38:29] src/van.cc:136: [00:38:29] src/van.cc:161: 1 => 1. Meta:
request=1, timestamp=4, control={ cmd=BARRIER, barrier_group=7 }
[00:38:29] src/van.cc:291: Barrier count for 7 : 3
? => 1. Meta: request=1, timestamp=4, control={ cmd=BARRIER, barrier_group=7
}
Now, it seems the program is blobked at somewhere, but I don't know how to
debug this problerm
I try to run "python train_mnist.py --network lenet --gpus 0 " on
10.15.133.82 in /tmp/mxnet, IT begin to train normally
but I run "python train_mnist.py --network lenet --gpus 0" on 10.15.240.189
in /tmp/mxnet, error occurs : src/van.cc:76: Check failed: (my_node_.port) !=
(-1) bind failed
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
