I tested the dynamic_percpu and it seems mostly works fine.
However, I found using dynamic_percpu in a constructor function leads to
general protection fault.
ex)
#include <stdio.h>
#include <assert.h>
#include <bsd/string.h>
#include <osv/preempt-lock.hh>
#include <osv/percpu.hh>
#include <osv/elf.hh>
struct counter {
int x = 0;
void inc(){
x += 1;
}
int get(){
return x;
}
};
dynamic_percpu<counter> c;
int main(int argc, char *argv[])
{
SCOPE_LOCK(preempt_lock);
c->inc();
return 0;
}
static __attribute__((constructor)) void test_init(void) {
SCOPE_LOCK(preempt_lock);
c->inc();
}
OSV_ELF_MLOCK_OBJECT();
output:
Cmdline: /hello
[registers]
RIP: 0x000010000000a8f8 <counter::inc()+12>
RFL: 0x0000000000010286 CS: 0x0000000000000008 SS: 0x0000000000000010
RAX: 0xffff7fffbf81b000 RBX: 0x0000000000000001 RCX: 0x0000000000000000
RDX: 0x0000000000000000
RSI: 0xffff7fffbf81b000 RDI: 0xffff7fffbf81b000 RBP: 0x0000200000200e70
R8: 0x0000000000000001
R9: 0x0000000000000000 R10: 0x000000000000000d R11: 0x0000000000000000
R12: 0xffffa000012bde00
R13: 0x0000000000000000 R14: 0x000000000000000d R15: 0x0000000000000000
RSP: 0x0000200000200e70
general protection fault
[backtrace]
0x00000000403a0e84 <general_protection+116>
0x000000004039e642 <???+1077536322>
0x000010000000a74f <???+42831>
0x000000004035546c <elf::object::run_init_funcs(int, char**)+268>
0x000000004035560a <elf::program::init_library(int, char**)+362>
0x0000000040224b7a <osv::application::main()+58>
0x000000004042a688 <???+1078109832>
0x0000000040462775 <???+1078339445>
0x00000000403faca6 <thread_main_c+38>
0x000000004039f602 <???+1077540354>
2020년 5월 26일 화요일 오후 11시 44분 23초 UTC+9, Wonsup Yoon 님의 말:
>
> Great! OSV_ELF_MLOCK_OBJECT seems very useful.
>
> Thanks.
>
>
>
> 2020년 5월 26일 화요일 오후 4시 25분 57초 UTC+9, Nadav Har'El 님의 말:
>>
>> On Tue, May 26, 2020 at 9:41 AM Wonsup Yoon <[email protected]> wrote:
>>
>>> Actually, I used preempt_lock to prevent data races.
>>> If two concurrent threads in a core access same per-cpu variable, I
>>> think we still need preempt lock.
>>>
>>
>> This is true - if you have two threads in the same core that access the
>> same per-cpu variable, you need some sort of locking.
>> preempt lock isn't the only way, of course - you can also use a mutex, as
>> well as std::atomic, and other solutions.
>>
>> preempt lock is indeed usually the fastest method, but as you saw it
>> comes with strings attached - the locked code really
>> cannot cause any preemption - which means it can't wait for any mutex,
>> cannot do anything (including delayed symbol
>> resolution which might wait for a mutex). In addition, you need to make
>> sure the entire object is already in memory and
>> doesn't need to be demand-paged, or you may get a preemption in the
>> middle of the code just to read in another page
>> of executable.
>>
>> We have a macro OSV_ELF_MLOCK_OBJECT() (from <osv/elf.hh>) which marks
>> the object with a flag (a
>> .note.osv-mlock section) that ensures *both* things: The object is
>> entirely read into memory on start, and all of
>> its symbols are resolved on start. You can see an example of
>> OSV_ELF_MLOCK_OBJECT() being used in a bunch
>> of tests in tests/. If you use this macro, you don't need to change your
>> code's compilation.
>>
>> example)
>>>
>>> counter's initial value: 0
>>>
>>> CPU 0
>>> Thread A A_local = counter + 1 (A_local = 1)
>>> Thread A *(preemption)*
>>> Thread B B_local = counter + 1 (B_local = 1)
>>> Thread B counter = B_local (counter = 1)
>>> Thread B *(exit)*
>>> Thread A counter = A_local (counter = 1)
>>>
>>> I expect counter to be 2, but 1 returns.
>>>
>>>
>>> 2020년 5월 26일 화요일 오후 3시 4분 24초 UTC+9, Nadav Har'El 님의 말:
>>>>
>>>>
>>>> On Tue, May 26, 2020 at 4:22 AM Wonsup Yoon <[email protected]> wrote:
>>>>
>>>>> Thank you for the response.
>>>>>
>>>>> Yes, dynamic_percpu<T> is perfect for my purpose.
>>>>>
>>>>> However, I encountered another issue.
>>>>>
>>>>> If I use dynamic_percpu with preempt-lock (I think it is very common
>>>>> pattern), it abort due to assertion failed.
>>>>> It seems lazy binding prevents preemption lock.
>>>>> So, I had to add -fno-plt option, and it works.
>>>>>
>>>>
>>>> You are right about preempt lock and your workaround for lazy binding.
>>>> However, to use a per-cpu variable, you don't need full preemption
>>>> locking - all you need is *migration* locking - in other words, the thread
>>>> running this code should not be migrated to a different CPU (this will
>>>> change the meaning of the per-cpu variable while you're using it), but it
>>>> is perfectly fine for the thread to be preempted to run a different thread
>>>> - as long as the original thread eventually returns to run on the same CPU
>>>> it previously ran on.
>>>>
>>>> So just replace your use of "preempt_lock" by "migration_lock" (include
>>>> <osv/migration-lock.hh>) and everything should work, without disabling
>>>> lazy
>>>> binding.
>>>>
>>>> Please note that if you use the per-cpu on a thread which is already
>>>> bound to a specific CPU (which was the case in your original code you
>>>> shared), you don't even need migration lock! A pinned thread already can't
>>>> migrate to any other CPU, so it doesn't need to use this
>>>> migration-avoidance mechanism at all. You can use per-cpu variables on
>>>> such
>>>> threads without any special protection.
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>> example code)
>>>>>
>>>>> #include <stdio.h>
>>>>> #include <assert.h>
>>>>>
>>>>> #include <osv/preempt-lock.hh>
>>>>> #include <osv/percpu.hh>
>>>>>
>>>>> struct counter {
>>>>> int x = 0;
>>>>>
>>>>> void inc(){
>>>>> x += 1;
>>>>> }
>>>>>
>>>>> int get(){
>>>>> return x;
>>>>> }
>>>>> };
>>>>>
>>>>> dynamic_percpu<counter> c;
>>>>>
>>>>> int main(int argc, char *argv[])
>>>>> {
>>>>> SCOPE_LOCK(preempt_lock);
>>>>> c->inc();
>>>>>
>>>>> return 0;
>>>>> }
>>>>>
>>>>>
>>>>> Backtrace)
>>>>>
>>>>> [backtrace]
>>>>> 0x000000004023875a <__assert_fail+26>
>>>>> 0x000000004035860c <elf::object::resolve_pltgot(unsigned int)+492>
>>>>> 0x0000000040358669 <elf_resolve_pltgot+57>
>>>>> 0x000000004039e2ef <???+1077535471>
>>>>> 0x000010000000f333 <???+62259>
>>>>> 0x000000004042a47c <osv::application::run_main()+60>
>>>>> 0x0000000040224bd0 <osv::application::main()+144>
>>>>> 0x000000004042a628 <???+1078109736>
>>>>> 0x0000000040462715 <???+1078339349>
>>>>> 0x00000000403fac86 <thread_main_c+38>
>>>>> 0x000000004039f632 <???+1077540402>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2020년 5월 24일 일요일 오후 5시 26분 17초 UTC+9, Nadav Har'El 님의 말:
>>>>>>
>>>>>>
>>>>>> On Sat, May 23, 2020 at 6:35 PM Wonsup Yoon <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm trying to use PERCPU macro in application or module.
>>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> The PERCPU macro does not support this. What it does is to add
>>>>>> information about this variable in a special section of the executable
>>>>>> (".percpu"), then arch/x64/loader.ld makes sure all these entries will
>>>>>> be
>>>>>> together between "_percpu_start" and "_percpu_end", and finally sched.cc
>>>>>> for every CPU creates (in the cpu::cpu(id) constructor) a copy of this
>>>>>> data. So if a loadable module (share library) contains another per-cpu
>>>>>> variable, it never gets added to the percpu area.
>>>>>>
>>>>>> However, I believe we do have a mechanism that will suite you:
>>>>>> *dynamic_percpu<T>*.
>>>>>> You can create (and destroy) such an object of type dynamic_percpu<T>
>>>>>> at any time, and it does the right thing: The variable will be
>>>>>> allocated
>>>>>> on all CPUs when the object is created, will be allocated on new cpus if
>>>>>> those happen, and will be freed when the object is destroyed.
>>>>>> In your case you can have a global dynamic_percpu<T> variable in your
>>>>>> loadable module. This object will be created when the module is loaded,
>>>>>> and
>>>>>> destroyed when the module is unloaded - which is what you want.
>>>>>>
>>>>>> Nadav.
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "OSv Development" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to [email protected].
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/osv-dev/07f76c69-0448-4a97-b587-995f7dbafe58%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/osv-dev/07f76c69-0448-4a97-b587-995f7dbafe58%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "OSv Development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to [email protected].
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/osv-dev/6f9dd47a-4f7a-46a8-89c4-fcaf1909dcc8%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/osv-dev/6f9dd47a-4f7a-46a8-89c4-fcaf1909dcc8%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/6ccc3dd5-9377-4372-9adf-7538c22eee71%40googlegroups.com.
