On Tue, May 10, 2016 at 3:17 AM, Savolainen, Petri (Nokia - FI/Espoo) < [email protected]> wrote:
> > > > > *From:* lng-odp [mailto:[email protected]] *On Behalf Of *Bill > Fischofer > *Sent:* Tuesday, May 10, 2016 12:54 AM > *To:* LNG ODP Mailman List <[email protected]> > *Subject:* [lng-odp] ODP Addressing Model > > > > The purpose of this thread is to summarize the discussions we've had over > the past week on this topic and to foster discussion. The goal here is to > reach consensus on a complete specification for ODP in this area that can > be added to the User Guide for Monarch, as well as to form the basis for > any expansion work needed in this area for Tiger Moth. > > > > Background > > ========= > > > > An ODP instance is bounded by everything between an odp_init_global() > through a matching odp_term_global() call. Applications are expected to > make use of a single ODP instance, though systems may support multiple > concurrent applications, each of which have their own independent ODP > instances. Communication between different applications is via independent > of whether either of them are making use of ODP and is expected to be via > standard OS or I/O mechanisms. The ODP Pktio IPC mechanism is experimental > and not enabled by default in Monarch. > > > > Threads and Processes > > ================== > > > > ODP itself does not define a threading model or provide APIs for creating > or destroying threads. The odp_thread_t type is intended to be an > abstraction of whatever underlying threading model is available to the > application via OS or other means outside the scope of ODP. ODP provides a > limited number of thread APIs for basic identification purposes. These > include: > > - odp_thread_id() - to obtain a unique thread ID for the current thread > - odp_thread_count() - to determine the number of threads that are > running in the current ODP instance > - odp_thread_count_max() - to determine the maximum number of threads > that can be supported by the current ODP instance > - odp_thread_type - to determine whether the current thread is a > worker or control thread > > In particular, ODP does not specify whether threads are implemented within > a single address space or separate address spaces. > > > > To facilitate operation in Linux environments, ODP provides a series of > "helper" APIs that layer on top of Linux pthreads. These are found in > helper/include/odp/helper/linux.h and include: > > - odph_linux_pthread_create() - to create a pthread > - odph_linux_pthread_join() - to wait for pthreads to exit > - odph_linux_process_fork() - to fork a linux process > - odph_linux_process_fork_n() - to fork a number of linux processes in > a single call > - odph_linux_ process_wait_n() - to wait for a number of linux > processes to exit > > These helper functions provide a means of setting pthread attributes and > process affinity flags as part of their operation, but again being helpers > and not ODP APIs they do not determine ODP API semantics. > > > > Issues > > ===== > > > > Several issues have been identified as a result of the above. The main > issue is what, if anything, does ODP have to say about whether addresses > derived from handles returned by ODP APIs may or may not be shared among > threads? The consensus seems to be that ODP itself is silent on this > subject. While it is recognized that it is convenient for applications to > share addresses between threads, whether or not they can do so safely is a > function of both application design and the underlying ODP implementation. > > > > For example, if an application consists of multiple threads prior to the > odp_global_init() call, then it is undefined whether handles, let alone > addresses, derived from that ODP instance are sharable with other > application threads. If the other application threads are pthreads sharing > the same address space, then perhaps yes, but if they are separate > processes then from the perspective of the ODP instance they are > effectively separate applications and hence things like ODP handles would > have no meaning within them. > > > > It seems that ODP should define the scope of an ODP instance (i.e., which > threads ODP handles derived from that instance are valid) to be the thread > that calls odp_init_global() and its direct descendants. > > > > Second, regarding addresses derived from ODP handles (e.g., those returned > by odp_shm_addr(), odp_packet_data(), etc.) again it seems that these can > be shared between threads only under two conditions: > > 1. The threads are within the same ODP instance > 2. The threads occupy a single shared address space or are forked from > the the same root thread that created the handle from which the returned > address is derived. > > Point 2 is again a function of both application design and the underlying > implementation. If the application creates all of its resources (pools, > shms, etc. during initialization prior to launching threads then these two > criteria should be satisfied). However, if the application starts creating > threads (particularly those that are not sharing the same address space) > and then those child threads are the ones that create pools, shms, etc., > then in general it does not seem safe to assume that addresses derived from > these handles have meaning to any other thread that is not a direct > descendent of the creating thread. > > > > From a portability and best practices standpoint it therefore seems that > we should encourage all global resource allocation to be performed by the > application during initialization after it has called odp_init_global() but > prior to creating other threads. > > > > I think this is too strict in general. Some implementation may be this > strict and some applications may be static enough to create all resources > in main thread at init time, but in general the recommendation should be > looser: > > > > · The “initialization” thread is the one calling odp_global_init() > > · The init thread creates all other ODP threads of an instance and > does that only after it has itself called odp_global_init() and > odp_local_init() > This seems like a reasonable recommendation, however we may wish to revisit this down the road to cover situations where additional resources are brought online to handle peak loads. For example, currently the number of CPUs available to an application is static and odp_cpumask_all_available() does not change. In a VNF environment one can imagine wanting to add additional processing resources to an existing VNF to handle peak loads and in that case more CPUs and associated worker threads would want to be brought up dynamically. Obviously we're not going to do that now, but whatever conventions we establish should scale to that sort of future more dynamic environment. > > > An implementation (under Linux) should allocate all internal shared memory > (e.g. memory for shm, pools, queues, etc resources and handles) (from > Linux) during global_init(). This way all ODP handles and pointers to ODP > managed memory may be shared between all ODP threads - regardless if > processes or pthreads are used as ODP threads. Naturally, all non-ODP > managed handles and memory are out of this scope (e.g. global variables of > an application are visible / not visible to other threads as specified by > the thread type). > I don't see how this is a less onerous assumption than the suggestion that all pools and shm's be allocated by the initial thread prior to creating other threads. First, it's hard to see how this suggestion can be implemented without API changes and since the Monarch API is frozen then any move in this direction would see to be post-Monarch. Also, as noted above, such a structure would seem to prevent future dynamic resource extensibility since applications would have to know in advance the upper limits of all resources they might require. Since pools and shms have differing attributes (shm_flags, or the different types of pools and attributes expressible via the odp_pool_param_t struct) this would seem to be at best an awkward requirement. We already have a precedent in the timers. We state that applications are expected to call odp_timer_pool_create() for each timer pool needed and then call odp_timer_pool_start(). I think the simplest, safest, and most portable rule to state is that ODP applications should assume that resources created by ODP APIs are created as shareable objects in the address space of the caller. If the application is using a single-address-space threading model (what odp-linux supports in Monarch) then it becomes automatic by this rule that ODP objects and their derived addresses would be sharable among all other threads sharing that address space. If a linux-style process model is being used, then again it's clear that sharing is possible if the resource is created prior to forking so that the shared object is present in both address spaces by standard Linux fork() semantics. If some other threading model is being used by some other implementation, then this rule would still be a useful portable design guideline for applications to follow. I don't see the need to stipulate beyond this (at least for Monarch). > > > User may pass maximum resource usage numbers (max shm usage, max pool > memory usage, max number of queues, etc) during build and/or run time > (global_init()). > > > > -Petri > > >
_______________________________________________ lng-odp mailing list [email protected] https://lists.linaro.org/mailman/listinfo/lng-odp
