Thanks for the patch. I'll merge it in with my patch. -Randy
On 2/7/13 11:25 AM, "Michael Robbert" <[email protected]> wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >Randy, >As long as you're working on IB patches. I just remembered that I had >to apply a patch before I could get 2.8.7 to build on my CentOS 5 >machines running their stock IB stack. > >- --- src/io/bmi/bmi_ib/openib.c.orig 2013-01-10 15:47:52.000000000 >- -0700 >+++ src/io/bmi/bmi_ib/openib.c 2013-01-10 15:37:59.000000000 -0700 >@@ -745,7 +745,9 @@ > #ifdef HAVE_IBV_EVENT_CLIENT_REREGISTER > CASE(IBV_EVENT_CLIENT_REREGISTER); > #endif >+#ifdef HAVE_IBV_EVENT_GID_CHANGE > CASE(IBV_EVENT_GID_CHANGE); >+#endif > } > return s; > } > >The issue was brought up in a thread on this list last summer, but I >never saw a final resolution and if there was one it apparently didn't >make it into 2.8.7 > >Thanks, >Mike Robbert >Colorado School of Mines > >On 2/7/13 8:20 AM, Randall Martin wrote: >> I'm working on a set of patches for the IB support. There are >> several issues I'm working through on the patches before I commit >> them. I'll send you a copy when I have them ready for release so >> you can test them. >> >> >> -Randy >> >> >> On 2/7/13 8:54 AM, "Yves Revaz" <[email protected]> wrote: >> >>> On 10/18/2012 11:41 PM, Kyle Schochenmaier wrote: >>>> Hi Yves - >>>> >>>> How frequently do you see these warnings? Does it cause any >>>> servers/clients to hang? >>> >>> Hi Kyle and the list, >>> >>> In a previous mail, I was mentioning the following errors: >>> >>> [E 02/07/2013 14:39:24] Warning: encourage_recv_incoming: mop_id >>> d0e680 in RTS_DONE message not found. [E 02/07/2013 14:39:54] >>> job_time_mgr_expire: job time out: cancelling flow operation, >>> job_id: 17549115350. [E 02/07/2013 14:39:54] >>> fp_multiqueue_cancel: flow proto cancel called on 0x1bce5e0 [E >>> 02/07/2013 14:39:54] fp_multiqueue_cancel: I/O error occurred [E >>> 02/07/2013 14:39:54] handle_io_error: flow proto error cleanup >>> started on 0x1bce5e0: Operation cancelled (possibly due to >>> timeout) [E 02/07/2013 14:39:54] handle_io_error: flow proto >>> 0x1bce5e0 canceled 1 operations, will clean up. [E 02/07/2013 >>> 14:39:54] bmi_recv_callback_fn: I/O error occurred [E 02/07/2013 >>> 14:39:54] handle_io_error: flow proto 0x1bce5e0 error cleanup >>> finished: Operation cancelled (possibly due to timeout) >>> >>> In fact, I'm trying to move 10Tb of data in our pvfs, using and >>> rsync. When a lot of data are transfered, those errors occurs >>> very frequently, about every 5 minutes, which is very annoying. >>> >>> I've checked our IB network which is perfectly sane. I'm >>> currently using orangefs-2.8.6/. Should I move to 2.8.7 ? Looking >>> at the changelog of the 2.8.7 realease, I don't thinks IB related >>> problems have been fixed. >>> >>> Thanks, >>> >>> yves >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>>> If not common/destructive this could be that there was a simple >>>> error case on the infiniband fabric and that the operation >>>> timed out in pvfs and that can be readily ignored as it would >>>> be retransmitted eventually. >>>> >>>> If you see this a lot it may be one of a few issues that we've >>>> fixed in recent releases, which version of orangefs/pvfs are >>>> you using? ~Kyle >>>> >>>> Kyle Schochenmaier >>>> >>>> >>>> On Thu, Oct 18, 2012 at 4:31 PM, Becky >>>> Ligon<[email protected]> wrote: >>>>> Yves: >>>>> >>>>> The timeouts that you listed below are in the configuration >>>>> file. >>>>> >>>>> ClientJobBMITimeoutSecs 300 - The client's job scheduler >>>>> limits each "job" sent across the network to this timeout. >>>>> If the job exceeds this limit, the job is cancelled. >>>>> Depending on the request, the job may be retried. Keep in >>>>> mind that one PVFS request can be made up of many jobs. >>>>> >>>>> ClientJobFlowTimeoutSecs - This value limits the time spent >>>>> on a particular job called a flow. A flow is used to >>>>> transfer data across the network to a server or to transfer >>>>> data from a server to the client. Again, if the flow >>>>> exceeds this timeout, then the flow is cancelled. >>>>> >>>>> The server counterparts for these settings are rarely used, >>>>> since the server doesn't normally initiate reads or writes. >>>>> >>>>> I think your real problem has something to do with IB, but I >>>>> am not an expert in that area. I have cc'd Kyle >>>>> Schochenmaier to see if he can help. >>>>> >>>>> Becky >>>>> >>>>> >>>>> >>>>> On Thu, Oct 18, 2012 at 4:07 PM, Yves >>>>> Revaz<[email protected]> wrote: >>>>>> >>>>>> Dear list, >>>>>> >>>>>> I sometimes have the following error occuring in my pvfs >>>>>> server log. >>>>>> >>>>>> [E 10/18/2012 20:59:50] Warning: encourage_recv_incoming: >>>>>> mop_id 150c320 in RTS_DONE message not found. [E 10/18/2012 >>>>>> 21:00:50] job_time_mgr_expire: job time out: cancelling >>>>>> flow operation, job_id: 33307291. [E 10/18/2012 21:00:50] >>>>>> fp_multiqueue_cancel: flow proto cancel called on 0xf18c80 >>>>>> [E 10/18/2012 21:00:50] fp_multiqueue_cancel: I/O error >>>>>> occurred [E 10/18/2012 21:00:50] handle_io_error: flow >>>>>> proto error cleanup started on 0xf18c80: Operation >>>>>> cancelled (possibly due to timeout) [E 10/18/2012 21:00:50] >>>>>> handle_io_error: flow proto 0xf18c80 canceled 1 operations, >>>>>> will clean up. [E 10/18/2012 21:00:50] >>>>>> bmi_recv_callback_fn: I/O error occurred [E 10/18/2012 >>>>>> 21:00:50] handle_io_error: flow proto 0xf18c80 error >>>>>> cleanup finished: Operation cancelled (possibly due to >>>>>> time >>>>>> >>>>>> >>>>>> Looking at the mailing list, I've found that increasing >>>>>> these default value (300) >>>>>> >>>>>> ServerJobBMITimeoutSecs 30 ServerJobFlowTimeoutSecs 30 >>>>>> ClientJobBMITimeoutSecs 300 ClientJobFlowTimeoutSecs 300 >>>>>> >>>>>> to 600. >>>>>> >>>>>> What is at the origin of these timeout ? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> >>>>>> yves >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- (o o) >>>>>> --------------------------------------------oOO--(_)--OOo------- >>>>>> >>>>>> >Dr. Yves Revaz >>>>>> Laboratory of Astrophysics EPFL >>>>>> >>>>>> Observatoire de Sauverny Tel : ++ 41 22 379 24 28 51. >>>>>> Ch. des Maillettes Fax : ++ 41 22 379 22 05 1290 >>>>>> Sauverny e-mail : [email protected] >>>>>> SWITZERLAND Web : >>>>>> http://www.lunix.ch/revaz/ >>>>>> ---------------------------------------------------------------- >>>>>> >>>>>> >>>>>> >_______________________________________________ >>>>>> Pvfs2-users mailing list >>>>>> [email protected] >>>>>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >>>>> >>>>> >>>>> >>>>> >>>>>> >- -- >>>>> Becky Ligon OrangeFS Support and Development Omnibond >>>>> Systems Anderson, South Carolina >>>>> >>>>> >>> >>> >>> -- >>> >>> ---------------------------------------------------------------- >>> Dr. Yves Revaz Laboratory of Astrophysics Ecole Polytechnique >>> F￩d￩rale de Lausanne (EPFL) Observatoire de Sauverny Tel : ++ >>> 41 22 379 24 28 51. Ch. des Maillettes Fax : ++ 41 22 379 >>> 22 05 1290 Sauverny e-mail : [email protected] >>> SWITZERLAND Web : http://www.lunix.ch/revaz/ >>> ---------------------------------------------------------------- >>> >>> _______________________________________________ Pvfs2-users >>> mailing list [email protected] >>> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >> >> >> >> _______________________________________________ Pvfs2-users mailing >> list [email protected] >> http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users >> >-----BEGIN PGP SIGNATURE----- >Version: GnuPG/MacGPG2 v2.0.19 (Darwin) >Comment: GPGTools - http://gpgtools.org > >iQEcBAEBAgAGBQJRE9VcAAoJEFmgPOBxQDtBEYMIAJtgo1LMWxVtyWPa2PNvWr2c >NMUw30GNJ2llhwJVdefpmNqPLdou0Sqr7moAPseA2qYBguER1jqSH0rnXg7yE5TX >CNERJwaL4+99y+tRsvKukrEvegrS/CQ5tUPsiuFaqqcTlQRGYeGPtqJV3JuAsEa2 >bu49sN7yWFtM2fY0ZaFa2ouya6PR2mFAdH0ZnpcWr4OTY1Uf4py8njWvvWrMCB/2 >I3//H5RoOxhCBIe85RCdXbMh4LMQbwBeTYFePlutE7YplbrQwDLg/K4/ctswRl3T >oKpRy5GJ83LJQomhwWWjAAnWWXe6zNlbiGe/B5APrlgZfV960shxFPeWwej3EEk= >=iXn7 >-----END PGP SIGNATURE----- >_______________________________________________ >Pvfs2-users mailing list >[email protected] >http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users _______________________________________________ Pvfs2-users mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
