Oh, it is not a OSv-specific problem.
It is because constructor function called before class initialization.
2020년 6월 2일 화요일 오후 11시 54분 26초 UTC+9, Wonsup Yoon 님의 말:
>
> I tested the dynamic_percpu and it seems mostly works fine.
>
> However, I found using dynamic_percpu in a constructor function leads to
> general protection fault.
>
> ex)
>
> #include <stdio.h>
> #include <assert.h>
> #include <bsd/string.h>
>
> #include <osv/preempt-lock.hh>
> #include <osv/percpu.hh>
> #include <osv/elf.hh>
>
> struct counter {
> int x = 0;
>
> void inc(){
> x += 1;
> }
>
> int get(){
> return x;
> }
> };
>
> dynamic_percpu<counter> c;
>
> int main(int argc, char *argv[])
> {
> SCOPE_LOCK(preempt_lock);
> c->inc();
>
> return 0;
> }
>
>
> static __attribute__((constructor)) void test_init(void) {
> SCOPE_LOCK(preempt_lock);
> c->inc();
> }
>
>
> OSV_ELF_MLOCK_OBJECT();
>
>
> output:
>
> Cmdline: /hello
>
> [registers]
>
> RIP: 0x000010000000a8f8 <counter::inc()+12>
>
> RFL: 0x0000000000010286 CS: 0x0000000000000008 SS: 0x0000000000000010
>
> RAX: 0xffff7fffbf81b000 RBX: 0x0000000000000001 RCX: 0x0000000000000000
> RDX: 0x0000000000000000
>
> RSI: 0xffff7fffbf81b000 RDI: 0xffff7fffbf81b000 RBP: 0x0000200000200e70
> R8: 0x0000000000000001
>
> R9: 0x0000000000000000 R10: 0x000000000000000d R11: 0x0000000000000000
> R12: 0xffffa000012bde00
>
> R13: 0x0000000000000000 R14: 0x000000000000000d R15: 0x0000000000000000
> RSP: 0x0000200000200e70
>
> general protection fault
>
>
> [backtrace]
>
> 0x00000000403a0e84 <general_protection+116>
>
> 0x000000004039e642 <???+1077536322>
>
> 0x000010000000a74f <???+42831>
>
> 0x000000004035546c <elf::object::run_init_funcs(int, char**)+268>
>
> 0x000000004035560a <elf::program::init_library(int, char**)+362>
>
> 0x0000000040224b7a <osv::application::main()+58>
>
> 0x000000004042a688 <???+1078109832>
>
> 0x0000000040462775 <???+1078339445>
>
> 0x00000000403faca6 <thread_main_c+38>
>
> 0x000000004039f602 <???+1077540354>
>
>
>
> 2020년 5월 26일 화요일 오후 11시 44분 23초 UTC+9, Wonsup Yoon 님의 말:
>>
>> Great! OSV_ELF_MLOCK_OBJECT seems very useful.
>>
>> Thanks.
>>
>>
>>
>> 2020년 5월 26일 화요일 오후 4시 25분 57초 UTC+9, Nadav Har'El 님의 말:
>>>
>>> On Tue, May 26, 2020 at 9:41 AM Wonsup Yoon <[email protected]> wrote:
>>>
>>>> Actually, I used preempt_lock to prevent data races.
>>>> If two concurrent threads in a core access same per-cpu variable, I
>>>> think we still need preempt lock.
>>>>
>>>
>>> This is true - if you have two threads in the same core that access the
>>> same per-cpu variable, you need some sort of locking.
>>> preempt lock isn't the only way, of course - you can also use a mutex,
>>> as well as std::atomic, and other solutions.
>>>
>>> preempt lock is indeed usually the fastest method, but as you saw it
>>> comes with strings attached - the locked code really
>>> cannot cause any preemption - which means it can't wait for any mutex,
>>> cannot do anything (including delayed symbol
>>> resolution which might wait for a mutex). In addition, you need to make
>>> sure the entire object is already in memory and
>>> doesn't need to be demand-paged, or you may get a preemption in the
>>> middle of the code just to read in another page
>>> of executable.
>>>
>>> We have a macro OSV_ELF_MLOCK_OBJECT() (from <osv/elf.hh>) which marks
>>> the object with a flag (a
>>> .note.osv-mlock section) that ensures *both* things: The object is
>>> entirely read into memory on start, and all of
>>> its symbols are resolved on start. You can see an example of
>>> OSV_ELF_MLOCK_OBJECT() being used in a bunch
>>> of tests in tests/. If you use this macro, you don't need to change your
>>> code's compilation.
>>>
>>> example)
>>>>
>>>> counter's initial value: 0
>>>>
>>>> CPU 0
>>>> Thread A A_local = counter + 1 (A_local = 1)
>>>> Thread A *(preemption)*
>>>> Thread B B_local = counter + 1 (B_local = 1)
>>>> Thread B counter = B_local (counter = 1)
>>>> Thread B *(exit)*
>>>> Thread A counter = A_local (counter = 1)
>>>>
>>>> I expect counter to be 2, but 1 returns.
>>>>
>>>>
>>>> 2020년 5월 26일 화요일 오후 3시 4분 24초 UTC+9, Nadav Har'El 님의 말:
>>>>>
>>>>>
>>>>> On Tue, May 26, 2020 at 4:22 AM Wonsup Yoon <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Thank you for the response.
>>>>>>
>>>>>> Yes, dynamic_percpu<T> is perfect for my purpose.
>>>>>>
>>>>>> However, I encountered another issue.
>>>>>>
>>>>>> If I use dynamic_percpu with preempt-lock (I think it is very common
>>>>>> pattern), it abort due to assertion failed.
>>>>>> It seems lazy binding prevents preemption lock.
>>>>>> So, I had to add -fno-plt option, and it works.
>>>>>>
>>>>>
>>>>> You are right about preempt lock and your workaround for lazy binding.
>>>>> However, to use a per-cpu variable, you don't need full preemption
>>>>> locking - all you need is *migration* locking - in other words, the
>>>>> thread
>>>>> running this code should not be migrated to a different CPU (this will
>>>>> change the meaning of the per-cpu variable while you're using it), but it
>>>>> is perfectly fine for the thread to be preempted to run a different
>>>>> thread
>>>>> - as long as the original thread eventually returns to run on the same
>>>>> CPU
>>>>> it previously ran on.
>>>>>
>>>>> So just replace your use of "preempt_lock" by "migration_lock"
>>>>> (include <osv/migration-lock.hh>) and everything should work, without
>>>>> disabling lazy binding.
>>>>>
>>>>> Please note that if you use the per-cpu on a thread which is already
>>>>> bound to a specific CPU (which was the case in your original code you
>>>>> shared), you don't even need migration lock! A pinned thread already
>>>>> can't
>>>>> migrate to any other CPU, so it doesn't need to use this
>>>>> migration-avoidance mechanism at all. You can use per-cpu variables on
>>>>> such
>>>>> threads without any special protection.
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> example code)
>>>>>>
>>>>>> #include <stdio.h>
>>>>>> #include <assert.h>
>>>>>>
>>>>>> #include <osv/preempt-lock.hh>
>>>>>> #include <osv/percpu.hh>
>>>>>>
>>>>>> struct counter {
>>>>>> int x = 0;
>>>>>>
>>>>>> void inc(){
>>>>>> x += 1;
>>>>>> }
>>>>>>
>>>>>> int get(){
>>>>>> return x;
>>>>>> }
>>>>>> };
>>>>>>
>>>>>> dynamic_percpu<counter> c;
>>>>>>
>>>>>> int main(int argc, char *argv[])
>>>>>> {
>>>>>> SCOPE_LOCK(preempt_lock);
>>>>>> c->inc();
>>>>>>
>>>>>> return 0;
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Backtrace)
>>>>>>
>>>>>> [backtrace]
>>>>>> 0x000000004023875a <__assert_fail+26>
>>>>>> 0x000000004035860c <elf::object::resolve_pltgot(unsigned int)+492>
>>>>>> 0x0000000040358669 <elf_resolve_pltgot+57>
>>>>>> 0x000000004039e2ef <???+1077535471>
>>>>>> 0x000010000000f333 <???+62259>
>>>>>> 0x000000004042a47c <osv::application::run_main()+60>
>>>>>> 0x0000000040224bd0 <osv::application::main()+144>
>>>>>> 0x000000004042a628 <???+1078109736>
>>>>>> 0x0000000040462715 <???+1078339349>
>>>>>> 0x00000000403fac86 <thread_main_c+38>
>>>>>> 0x000000004039f632 <???+1077540402>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2020년 5월 24일 일요일 오후 5시 26분 17초 UTC+9, Nadav Har'El 님의 말:
>>>>>>>
>>>>>>>
>>>>>>> On Sat, May 23, 2020 at 6:35 PM Wonsup Yoon <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm trying to use PERCPU macro in application or module.
>>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> The PERCPU macro does not support this. What it does is to add
>>>>>>> information about this variable in a special section of the executable
>>>>>>> (".percpu"), then arch/x64/loader.ld makes sure all these entries will
>>>>>>> be
>>>>>>> together between "_percpu_start" and "_percpu_end", and finally
>>>>>>> sched.cc
>>>>>>> for every CPU creates (in the cpu::cpu(id) constructor) a copy of this
>>>>>>> data. So if a loadable module (share library) contains another per-cpu
>>>>>>> variable, it never gets added to the percpu area.
>>>>>>>
>>>>>>> However, I believe we do have a mechanism that will suite you:
>>>>>>> *dynamic_percpu<T>*.
>>>>>>> You can create (and destroy) such an object of type
>>>>>>> dynamic_percpu<T> at any time, and it does the right thing: The
>>>>>>> variable
>>>>>>> will be allocated on all CPUs when the object is created, will be
>>>>>>> allocated
>>>>>>> on new cpus if those happen, and will be freed when the object is
>>>>>>> destroyed.
>>>>>>> In your case you can have a global dynamic_percpu<T> variable in
>>>>>>> your loadable module. This object will be created when the module is
>>>>>>> loaded, and destroyed when the module is unloaded - which is what you
>>>>>>> want.
>>>>>>>
>>>>>>> Nadav.
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "OSv Development" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to [email protected].
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/osv-dev/07f76c69-0448-4a97-b587-995f7dbafe58%40googlegroups.com
>>>>>>
>>>>>> <https://groups.google.com/d/msgid/osv-dev/07f76c69-0448-4a97-b587-995f7dbafe58%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>>> .
>>>>>>
>>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "OSv Development" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/osv-dev/6f9dd47a-4f7a-46a8-89c4-fcaf1909dcc8%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/osv-dev/6f9dd47a-4f7a-46a8-89c4-fcaf1909dcc8%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/4a820afe-655a-42a6-86f2-41bd36903972%40googlegroups.com.
