I think this is going to need to be refactored, Josh. Looks to me like whoever wrote this "locked" it to IB. That's a problem for me, at least - I believe that OpenShmem should use the MPI transports, but respect the OMPI abstraction layer.
In other words, you should use the transports to move data around, but let the underlying abstractions choose how to do it. This would allow OpenShmem to run on top of any fabric, not just IB, which is the right way of doing things inside the OMPI code base. Just my $0.02 Ralph On Jul 24, 2013, at 5:03 AM, Ralph Castain <r...@open-mpi.org> wrote: > Hmmmm....I'm afraid this doesn't build for us folks that don't have IB on > their system. From what I can see, it looks like you broke a fundamental > abstraction layer rule by including a header from a component into a > completely different component. That is definitely *not* allowed for > precisely this reason. > > In file included from ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:32, > from ../../../oshmem/mca/spml/yoda/spml_yoda_rdmafrag.h:21, > from ../../../oshmem/mca/spml/yoda/spml_yoda_putreq.h:19, > from base/spml_base.c:19: > ../../../ompi/mca/btl/openib/btl_openib.h:34:30: error: infiniband/verbs.h: > No such file or directory > In file included from ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:32, > from ../../../oshmem/mca/spml/yoda/spml_yoda_rdmafrag.h:21, > from ../../../oshmem/mca/spml/yoda/spml_yoda_putreq.h:19, > from base/spml_base.c:19: > ../../../ompi/mca/btl/openib/btl_openib.h:374: error: field 'ib_dev_attr' has > incomplete type > In file included from ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:32, > from ../../../oshmem/mca/spml/yoda/spml_yoda_rdmafrag.h:21, > from ../../../oshmem/mca/spml/yoda/spml_yoda_putreq.h:19, > from base/spml_base.c:19: > ../../../ompi/mca/btl/openib/btl_openib.h:470: error: field 'ib_port_attr' > has incomplete type > In file included from ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:33, > from ../../../oshmem/mca/spml/yoda/spml_yoda_rdmafrag.h:21, > from ../../../oshmem/mca/spml/yoda/spml_yoda_putreq.h:19, > from base/spml_base.c:19: > ../../../ompi/mca/btl/openib/btl_openib_frag.h:306: error: field 'sg_entry' > has incomplete type > ../../../ompi/mca/btl/openib/btl_openib_frag.h:318: error: field 'sr_desc' > has incomplete type > ../../../ompi/mca/btl/openib/btl_openib_frag.h:345: error: field 'rd_desc' > has incomplete type > ../../../ompi/mca/btl/openib/btl_openib_frag.h:359: error: field 'sr_desc' > has incomplete type > In file included from ../../../oshmem/mca/spml/yoda/spml_yoda_rdmafrag.h:21, > from ../../../oshmem/mca/spml/yoda/spml_yoda_putreq.h:19, > from base/spml_base.c:19: > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h: In function 'post_recvs': > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:367: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:368: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:368: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:368: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:368: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:368: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:376: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:378: error: implicit > declaration of function 'ibv_post_recv' > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h: In function > 'ib_send_flags': > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:515: error: > 'IBV_SEND_SIGNALED' undeclared (first use in this function) > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:515: error: (Each > undeclared identifier is reported only once > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:515: error: for each > function it appears in.) > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:516: error: > 'IBV_SEND_INLINE' undeclared (first use in this function) > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h: In function 'post_send': > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:543: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:546: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:546: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:554: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:556: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:556: error: > 'IBV_WR_RDMA_WRITE' undeclared (first use in this function) > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:557: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:569: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:578: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:584: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:584: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:587: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:587: error: 'IBV_WR_SEND' > undeclared (first use in this function) > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:589: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:589: error: > 'IBV_WR_SEND_WITH_IMM' undeclared (first use in this function) > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:593: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:602: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:604: error: dereferencing > pointer to incomplete type > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:604: error: > 'IBV_SEND_SIGNALED' undeclared (first use in this function) > ../../../ompi/mca/btl/openib/btl_openib_endpoint.h:610: error: implicit > declaration of function 'ibv_post_send' > make[2]: *** [base/spml_base.lo] Error 1 > make[2]: *** Waiting for unfinished jobs.... > make[1]: *** [all-recursive] Error 1 > make: *** [all-recursive] Error 1 > > > > On Jul 24, 2013, at 4:51 AM, Ralph Castain <r...@open-mpi.org> wrote: > >> I'm afraid Josh got bit by grabbing the wrong URL - here's the correct clone >> command for anyone not Josh: >> >> git clone https://bitbucket.org/jladd_math/mlnx-oshmem.git >> >> >> On Jul 23, 2013, at 6:43 AM, Joshua Ladd <josh...@mellanox.com> wrote: >> >>> Dear OMPI Developers, >>> >>> I have put Mellanox OpenSHMEM up for review on my Bitbucket. Please “git” >>> and test at your leisure. Questions, comments, and critiques are most >>> welcome. >>> >>> git clone https://jladd_m...@bitbucket.org/jladd_math/mlnx-oshmem.git >>> >>> To build with OSHMEM support, build as you would OMPI but simply include >>> ‘--with-oshmem’ on your configure line. This will get you started. >>> >>> Best regards, >>> >>> Josh >>> >>> >>> >>> Joshua S. Ladd, PhD >>> HPC Algorithms Engineer >>> Mellanox Technologies >>> >>> Email: josh...@mellanox.com >>> Cell: +1 (865) 258 - 8898 >>> >>> >>> _______________________________________________ >>> devel mailing list >>> de...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> >
