Great! OSV_ELF_MLOCK_OBJECT seems very useful. Thanks.
2020년 5월 26일 화요일 오후 4시 25분 57초 UTC+9, Nadav Har'El 님의 말: > > On Tue, May 26, 2020 at 9:41 AM Wonsup Yoon <[email protected] > <javascript:>> wrote: > >> Actually, I used preempt_lock to prevent data races. >> If two concurrent threads in a core access same per-cpu variable, I think >> we still need preempt lock. >> > > This is true - if you have two threads in the same core that access the > same per-cpu variable, you need some sort of locking. > preempt lock isn't the only way, of course - you can also use a mutex, as > well as std::atomic, and other solutions. > > preempt lock is indeed usually the fastest method, but as you saw it comes > with strings attached - the locked code really > cannot cause any preemption - which means it can't wait for any mutex, > cannot do anything (including delayed symbol > resolution which might wait for a mutex). In addition, you need to make > sure the entire object is already in memory and > doesn't need to be demand-paged, or you may get a preemption in the middle > of the code just to read in another page > of executable. > > We have a macro OSV_ELF_MLOCK_OBJECT() (from <osv/elf.hh>) which marks > the object with a flag (a > .note.osv-mlock section) that ensures *both* things: The object is > entirely read into memory on start, and all of > its symbols are resolved on start. You can see an example of > OSV_ELF_MLOCK_OBJECT() being used in a bunch > of tests in tests/. If you use this macro, you don't need to change your > code's compilation. > > example) >> >> counter's initial value: 0 >> >> CPU 0 >> Thread A A_local = counter + 1 (A_local = 1) >> Thread A *(preemption)* >> Thread B B_local = counter + 1 (B_local = 1) >> Thread B counter = B_local (counter = 1) >> Thread B *(exit)* >> Thread A counter = A_local (counter = 1) >> >> I expect counter to be 2, but 1 returns. >> >> >> 2020년 5월 26일 화요일 오후 3시 4분 24초 UTC+9, Nadav Har'El 님의 말: >>> >>> >>> On Tue, May 26, 2020 at 4:22 AM Wonsup Yoon <[email protected]> wrote: >>> >>>> Thank you for the response. >>>> >>>> Yes, dynamic_percpu<T> is perfect for my purpose. >>>> >>>> However, I encountered another issue. >>>> >>>> If I use dynamic_percpu with preempt-lock (I think it is very common >>>> pattern), it abort due to assertion failed. >>>> It seems lazy binding prevents preemption lock. >>>> So, I had to add -fno-plt option, and it works. >>>> >>> >>> You are right about preempt lock and your workaround for lazy binding. >>> However, to use a per-cpu variable, you don't need full preemption >>> locking - all you need is *migration* locking - in other words, the thread >>> running this code should not be migrated to a different CPU (this will >>> change the meaning of the per-cpu variable while you're using it), but it >>> is perfectly fine for the thread to be preempted to run a different thread >>> - as long as the original thread eventually returns to run on the same CPU >>> it previously ran on. >>> >>> So just replace your use of "preempt_lock" by "migration_lock" (include >>> <osv/migration-lock.hh>) and everything should work, without disabling lazy >>> binding. >>> >>> Please note that if you use the per-cpu on a thread which is already >>> bound to a specific CPU (which was the case in your original code you >>> shared), you don't even need migration lock! A pinned thread already can't >>> migrate to any other CPU, so it doesn't need to use this >>> migration-avoidance mechanism at all. You can use per-cpu variables on such >>> threads without any special protection. >>> >>> >>>> >>>> >>>> >>>> example code) >>>> >>>> #include <stdio.h> >>>> #include <assert.h> >>>> >>>> #include <osv/preempt-lock.hh> >>>> #include <osv/percpu.hh> >>>> >>>> struct counter { >>>> int x = 0; >>>> >>>> void inc(){ >>>> x += 1; >>>> } >>>> >>>> int get(){ >>>> return x; >>>> } >>>> }; >>>> >>>> dynamic_percpu<counter> c; >>>> >>>> int main(int argc, char *argv[]) >>>> { >>>> SCOPE_LOCK(preempt_lock); >>>> c->inc(); >>>> >>>> return 0; >>>> } >>>> >>>> >>>> Backtrace) >>>> >>>> [backtrace] >>>> 0x000000004023875a <__assert_fail+26> >>>> 0x000000004035860c <elf::object::resolve_pltgot(unsigned int)+492> >>>> 0x0000000040358669 <elf_resolve_pltgot+57> >>>> 0x000000004039e2ef <???+1077535471> >>>> 0x000010000000f333 <???+62259> >>>> 0x000000004042a47c <osv::application::run_main()+60> >>>> 0x0000000040224bd0 <osv::application::main()+144> >>>> 0x000000004042a628 <???+1078109736> >>>> 0x0000000040462715 <???+1078339349> >>>> 0x00000000403fac86 <thread_main_c+38> >>>> 0x000000004039f632 <???+1077540402> >>>> >>>> >>>> >>>> >>>> 2020년 5월 24일 일요일 오후 5시 26분 17초 UTC+9, Nadav Har'El 님의 말: >>>>> >>>>> >>>>> On Sat, May 23, 2020 at 6:35 PM Wonsup Yoon <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm trying to use PERCPU macro in application or module. >>>>>> >>>>> >>>>> Hi, >>>>> >>>>> The PERCPU macro does not support this. What it does is to add >>>>> information about this variable in a special section of the executable >>>>> (".percpu"), then arch/x64/loader.ld makes sure all these entries will be >>>>> together between "_percpu_start" and "_percpu_end", and finally sched.cc >>>>> for every CPU creates (in the cpu::cpu(id) constructor) a copy of this >>>>> data. So if a loadable module (share library) contains another per-cpu >>>>> variable, it never gets added to the percpu area. >>>>> >>>>> However, I believe we do have a mechanism that will suite you: >>>>> *dynamic_percpu<T>*. >>>>> You can create (and destroy) such an object of type dynamic_percpu<T> >>>>> at any time, and it does the right thing: The variable will be allocated >>>>> on all CPUs when the object is created, will be allocated on new cpus if >>>>> those happen, and will be freed when the object is destroyed. >>>>> In your case you can have a global dynamic_percpu<T> variable in your >>>>> loadable module. This object will be created when the module is loaded, >>>>> and >>>>> destroyed when the module is unloaded - which is what you want. >>>>> >>>>> Nadav. >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "OSv Development" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/osv-dev/07f76c69-0448-4a97-b587-995f7dbafe58%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/osv-dev/07f76c69-0448-4a97-b587-995f7dbafe58%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "OSv Development" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/osv-dev/6f9dd47a-4f7a-46a8-89c4-fcaf1909dcc8%40googlegroups.com >> >> <https://groups.google.com/d/msgid/osv-dev/6f9dd47a-4f7a-46a8-89c4-fcaf1909dcc8%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/cbd8488d-2f4c-438f-928b-0325c4d1107e%40googlegroups.com.
