Hello/bonjour,
I modified your patch so that in the worse case exactly comm->num_procs are
probed (I just changed a < for a <= in the second for loop).
The behavior of your patch was to probe a maximum of comm->num_procs - 1, thus
hanging if comm->last_probed has to be probed.
With the patch, I get a few good time slices, but most are not productive, see
below.
Therefore I strongly think that you were right to say that my program should
not use MPI_ANY_SOURCE in the first place.
I'll stick with my round-robin reception in which many calls to MPI_Iprobe says
"no message".
In my round-robin reception, I guess that the opal_progress() call in
mca_pml_ob1_iprobe (ompi/mca/pml/ob1/pml_ob1_iprobe.c) for all the calls
to MPI_Iprobe with no match is *really* what helps draining the messages and
presumably contributing to the fairness of the whole thing.
What do you think ?
If I run Ray in gdb:
(gdb) run
Starting program: /home/sboisver12/git-clones/ray/Ray
[Thread debugging using libthread_db enabled]
[New Thread 0x40a00940 (LWP 16867)]
[New Thread 0x41401940 (LWP 16869)]
I see two new threads that are spawned.
The draining of messages is also done in a "progress thread", right ?
Thank you.
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.47 s Speed= 90354 Sent= 30
(processMessages: 30, processData: 0) Received= 30 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.57 s Speed= 93748 Sent= 26
(processMessages: 26, processData: 0) Received= 26 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.67 s Speed= 92086 Sent= 66
(processMessages: 66, processData: 0) Received= 66 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.77 s Speed= 93501 Sent= 104
(processMessages: 104, processData: 0) Received= 104 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.87 s Speed= 94512 Sent= 85
(processMessages: 85, processData: 0) Received= 85 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4176.97 s Speed= 93262 Sent= 138
(processMessages: 138, processData: 0) Received= 138 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.07 s Speed= 94944 Sent= 125
(processMessages: 125, processData: 0) Received= 125 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.17 s Speed= 95377 Sent= 119
(processMessages: 119, processData: 0) Received= 119 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.27 s Speed= 91206 Sent= 129
(processMessages: 129, processData: 0) Received= 129 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.37 s Speed= 92937 Sent= 83
(processMessages: 83, processData: 0) Received= 83 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.47 s Speed= 90803 Sent= 142
(processMessages: 142, processData: 0) Received= 142 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.57 s Speed= 94807 Sent= 87
(processMessages: 87, processData: 0) Received= 87 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.67 s Speed= 92527 Sent= 176
(processMessages: 176, processData: 0) Received= 176 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.77 s Speed= 93514 Sent= 151
(processMessages: 151, processData: 0) Received= 151 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.87 s Speed= 94102 Sent= 81
(processMessages: 81, processData: 0) Received= 81 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4177.97 s Speed= 95072 Sent= 104
(processMessages: 104, processData: 0) Received= 104 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.07 s Speed= 90934 Sent= 99
(processMessages: 99, processData: 0) Received= 99 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.17 s Speed= 92640 Sent= 117
(processMessages: 117, processData: 0) Received= 117 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.27 s Speed= 93260 Sent= 84
(processMessages: 84, processData: 0) Received= 84 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.37 s Speed= 94774 Sent= 134
(processMessages: 134, processData: 0) Received= 134 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.47 s Speed= 94714 Sent= 84
(processMessages: 84, processData: 0) Received= 84 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.57 s Speed= 94921 Sent= 55
(processMessages: 55, processData: 0) Received= 55 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.67 s Speed= 94461 Sent= 48
(processMessages: 48, processData: 0) Received= 48 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.77 s Speed= 93824 Sent= 58
(processMessages: 58, processData: 0) Received= 58 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.87 s Speed= 90685 Sent= 131
(processMessages: 131, processData: 0) Received= 131 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4178.97 s Speed= 92930 Sent= 65
(processMessages: 65, processData: 0) Received= 65 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.07 s Speed= 93235 Sent= 38
(processMessages: 38, processData: 0) Received= 38 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.17 s Speed= 92487 Sent= 56
(processMessages: 56, processData: 0) Received= 56 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.27 s Speed= 92559 Sent= 61
(processMessages: 61, processData: 0) Received= 61 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.37 s Speed= 91398 Sent= 47
(processMessages: 47, processData: 0) Received= 47 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.47 s Speed= 94528 Sent= 72
(processMessages: 72, processData: 0) Received= 72 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.57 s Speed= 93548 Sent= 112
(processMessages: 112, processData: 0) Received= 112 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.67 s Speed= 94701 Sent= 54
(processMessages: 54, processData: 0) Received= 54 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.77 s Speed= 94900 Sent= 50
(processMessages: 50, processData: 0) Received= 50 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.87 s Speed= 92868 Sent= 43
(processMessages: 43, processData: 0) Received= 43 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4179.97 s Speed= 92996 Sent= 170
(processMessages: 54, processData: 116) Received= 170 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.07 s Speed= 88456 Sent= 93
(processMessages: 77, processData: 16) Received= 93 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.17 s Speed= 88194 Sent= 874
(processMessages: 83, processData: 791) Received= 874 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.27 s Speed= 84966 Sent= 439
(processMessages: 79, processData: 360) Received= 439 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.37 s Speed= 83100 Sent= 406
(processMessages: 67, processData: 339) Received= 406 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.47 s Speed= 88243 Sent= 215
(processMessages: 54, processData: 161) Received= 215 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.57 s Speed= 87354 Sent= 550
(processMessages: 201, processData: 349) Received= 550 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.67 s Speed= 94519 Sent= 55
(processMessages: 55, processData: 0) Received= 55 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.77 s Speed= 90221 Sent= 79
(processMessages: 79, processData: 0) Received= 79 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.87 s Speed= 92418 Sent= 146
(processMessages: 146, processData: 0) Received= 146 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4180.97 s Speed= 94045 Sent= 68
(processMessages: 68, processData: 0) Received= 68 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.07 s Speed= 94432 Sent= 64
(processMessages: 64, processData: 0) Received= 64 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.17 s Speed= 90937 Sent= 68
(processMessages: 68, processData: 0) Received= 68 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.27 s Speed= 88159 Sent= 62
(processMessages: 62, processData: 0) Received= 62 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.37 s Speed= 94526 Sent= 84
(processMessages: 84, processData: 0) Received= 84 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.47 s Speed= 93362 Sent= 73
(processMessages: 73, processData: 0) Received= 73 Balance= 0
Rank 255: RAY_SLAVE_MODE_EXTENSION Time= 4181.57 s Speed= 90062 Sent= 45
(processMessages: 45, processData: 0) Received= 45 Balance= 0
> ________________________________________
> De : devel-boun...@open-mpi.org [devel-boun...@open-mpi.org] de la part de
> George Bosilca [bosi...@eecs.utk.edu]
> Date d'envoi : 27 septembre 2011 16:44
> À : Open MPI Developers
> Objet : Re: [OMPI devel] RE : Implementation of MPI_Iprobe
>
> Sebastien,
>
> Please try the attached patch. It is made against the trunk, so you might
> have to adapt it slightly.
>
> Let me know of the outcome, so we can decide if it's worth pushing it in the
> next releases.
>
> Thanks,
> george.
>
>
SébastienIndex: pml_ob1_comm.c
===================================================================
--- pml_ob1_comm.c (revision 25195)
+++ pml_ob1_comm.c (working copy)
@@ -56,6 +56,7 @@
OBJ_CONSTRUCT(&comm->matching_lock, opal_mutex_t);
comm->recv_sequence = 0;
comm->procs = NULL;
+ comm->last_probed = 0;
comm->num_procs = 0;
}
Index: pml_ob1_comm.h
===================================================================
--- pml_ob1_comm.h (revision 25195)
+++ pml_ob1_comm.h (working copy)
@@ -58,6 +58,7 @@
opal_list_t wild_receives; /**< queue of unmatched wild (source process not specified) receives */
mca_pml_ob1_comm_proc_t* procs;
size_t num_procs;
+ size_t last_probed;
};
typedef struct mca_pml_comm_t mca_pml_ob1_comm_t;
Index: pml_ob1_recvreq.c
===================================================================
--- pml_ob1_recvreq.c (revision 25195)
+++ pml_ob1_recvreq.c (working copy)
@@ -900,19 +900,34 @@
* There is an outer loop over lists of messages from each
* process, then an inner loop over the messages from the
* process.
+ *
+ * In order to avoid starvation do this in a round-robin fashion.
*/
- for (i = 0; i < proc_count; i++) {
+ for (i = comm->last_probed + 1; i < comm->num_procs; i++) {
mca_pml_ob1_recv_frag_t* frag;
/* loop over messages from the current proc */
if((frag = recv_req_match_specific_proc(req, &proc[i]))) {
*p = &proc[i];
+ comm->last_probed = i;
req->req_recv.req_base.req_proc = proc[i].ompi_proc;
prepare_recv_req_converter(req);
return frag; /* match found */
}
}
+ for (i = 0; i <= comm->last_probed; i++) {
+ mca_pml_ob1_recv_frag_t* frag;
+ /* loop over messages from the current proc */
+ if((frag = recv_req_match_specific_proc(req, &proc[i]))) {
+ *p = &proc[i];
+ comm->last_probed = i;
+ req->req_recv.req_base.req_proc = proc[i].ompi_proc;
+ prepare_recv_req_converter(req);
+ return frag; /* match found */
+ }
+ }
+
*p = NULL;
return NULL;
}
