From: lng-odp [mailto:[email protected]] On Behalf Of Bill Fischofer Sent: Tuesday, May 10, 2016 12:54 AM To: LNG ODP Mailman List <[email protected]> Subject: [lng-odp] ODP Addressing Model
The purpose of this thread is to summarize the discussions we've had over the past week on this topic and to foster discussion. The goal here is to reach consensus on a complete specification for ODP in this area that can be added to the User Guide for Monarch, as well as to form the basis for any expansion work needed in this area for Tiger Moth. Background ========= An ODP instance is bounded by everything between an odp_init_global() through a matching odp_term_global() call. Applications are expected to make use of a single ODP instance, though systems may support multiple concurrent applications, each of which have their own independent ODP instances. Communication between different applications is via independent of whether either of them are making use of ODP and is expected to be via standard OS or I/O mechanisms. The ODP Pktio IPC mechanism is experimental and not enabled by default in Monarch. Threads and Processes ================== ODP itself does not define a threading model or provide APIs for creating or destroying threads. The odp_thread_t type is intended to be an abstraction of whatever underlying threading model is available to the application via OS or other means outside the scope of ODP. ODP provides a limited number of thread APIs for basic identification purposes. These include: * odp_thread_id() - to obtain a unique thread ID for the current thread * odp_thread_count() - to determine the number of threads that are running in the current ODP instance * odp_thread_count_max() - to determine the maximum number of threads that can be supported by the current ODP instance * odp_thread_type - to determine whether the current thread is a worker or control thread In particular, ODP does not specify whether threads are implemented within a single address space or separate address spaces. To facilitate operation in Linux environments, ODP provides a series of "helper" APIs that layer on top of Linux pthreads. These are found in helper/include/odp/helper/linux.h and include: * odph_linux_pthread_create() - to create a pthread * odph_linux_pthread_join() - to wait for pthreads to exit * odph_linux_process_fork() - to fork a linux process * odph_linux_process_fork_n() - to fork a number of linux processes in a single call * odph_linux_ process_wait_n() - to wait for a number of linux processes to exit These helper functions provide a means of setting pthread attributes and process affinity flags as part of their operation, but again being helpers and not ODP APIs they do not determine ODP API semantics. Issues ===== Several issues have been identified as a result of the above. The main issue is what, if anything, does ODP have to say about whether addresses derived from handles returned by ODP APIs may or may not be shared among threads? The consensus seems to be that ODP itself is silent on this subject. While it is recognized that it is convenient for applications to share addresses between threads, whether or not they can do so safely is a function of both application design and the underlying ODP implementation. For example, if an application consists of multiple threads prior to the odp_global_init() call, then it is undefined whether handles, let alone addresses, derived from that ODP instance are sharable with other application threads. If the other application threads are pthreads sharing the same address space, then perhaps yes, but if they are separate processes then from the perspective of the ODP instance they are effectively separate applications and hence things like ODP handles would have no meaning within them. It seems that ODP should define the scope of an ODP instance (i.e., which threads ODP handles derived from that instance are valid) to be the thread that calls odp_init_global() and its direct descendants. Second, regarding addresses derived from ODP handles (e.g., those returned by odp_shm_addr(), odp_packet_data(), etc.) again it seems that these can be shared between threads only under two conditions: 1. The threads are within the same ODP instance 2. The threads occupy a single shared address space or are forked from the the same root thread that created the handle from which the returned address is derived. Point 2 is again a function of both application design and the underlying implementation. If the application creates all of its resources (pools, shms, etc. during initialization prior to launching threads then these two criteria should be satisfied). However, if the application starts creating threads (particularly those that are not sharing the same address space) and then those child threads are the ones that create pools, shms, etc., then in general it does not seem safe to assume that addresses derived from these handles have meaning to any other thread that is not a direct descendent of the creating thread. From a portability and best practices standpoint it therefore seems that we should encourage all global resource allocation to be performed by the application during initialization after it has called odp_init_global() but prior to creating other threads. I think this is too strict in general. Some implementation may be this strict and some applications may be static enough to create all resources in main thread at init time, but in general the recommendation should be looser: · The “initialization” thread is the one calling odp_global_init() · The init thread creates all other ODP threads of an instance and does that only after it has itself called odp_global_init() and odp_local_init() An implementation (under Linux) should allocate all internal shared memory (e.g. memory for shm, pools, queues, etc resources and handles) (from Linux) during global_init(). This way all ODP handles and pointers to ODP managed memory may be shared between all ODP threads - regardless if processes or pthreads are used as ODP threads. Naturally, all non-ODP managed handles and memory are out of this scope (e.g. global variables of an application are visible / not visible to other threads as specified by the thread type). User may pass maximum resource usage numbers (max shm usage, max pool memory usage, max number of queues, etc) during build and/or run time (global_init()). -Petri
_______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
