I tested the dynamic_percpu and it seems mostly works fine.
However, I found using dynamic_percpu in a constructor function leads to
general protection fault.
ex)
#include <stdio.h>
#include <assert.h>
#include <bsd/string.h>
#include <osv/preempt-lock.hh>
#include <osv/percpu.hh>
#include <osv/elf.hh>
struct counter {
int x = 0;
void inc(){
x += 1;
}
int get(){
return x;
}
};
dynamic_percpu<counter> c;
int main(int argc, char *argv[])
{
SCOPE_LOCK(preempt_lock);
c->inc();
return 0;
}
static __attribute__((constructor)) void test_init(void) {
SCOPE_LOCK(preempt_lock);
c->inc();
}
OSV_ELF_MLOCK_OBJECT();
output:
Cmdline: /hello
[registers]
RIP: 0x000010000000a8f8 <counter::inc()+12>
RFL: 0x0000000000010286 CS: 0x0000000000000008 SS: 0x0000000000000010
RAX: 0xffff7fffbf81b000 RBX: 0x0000000000000001 RCX: 0x0000000000000000
RDX: 0x0000000000000000
RSI: 0xffff7fffbf81b000 RDI: 0xffff7fffbf81b000 RBP: 0x0000200000200e70
R8: 0x0000000000000001
R9: 0x0000000000000000 R10: 0x000000000000000d R11: 0x0000000000000000
R12: 0xffffa000012bde00
R13: 0x0000000000000000 R14: 0x000000000000000d R15: 0x0000000000000000
RSP: 0x0000200000200e70
general protection fault
[backtrace]
0x00000000403a0e84 <general_protection+116>
0x000000004039e642 <???+1077536322>
0x000010000000a74f <???+42831>
0x000000004035546c <elf::object::run_init_funcs(int, char**)+268>
0x000000004035560a <elf::program::init_library(int, char**)+362>
0x0000000040224b7a <osv::application::main()+58>
0x000000004042a688 <???+1078109832>
0x0000000040462775 <???+1078339445>
0x00000000403faca6 <thread_main_c+38>
0x000000004039f602 <???+1077540354>
2020년 5월 26일 화요일 오후 11시 44분 23초 UTC+9, Wonsup Yoon 님의 말:
>
> Great! OSV_ELF_MLOCK_OBJECT seems very useful.
>
> Thanks.
>
>
>
> 2020년 5월 26일 화요일 오후 4시 25분 57초 UTC+9, Nadav Har'El 님의 말:
>>
>> On Tue, May 26, 2020 at 9:41 AM Wonsup Yoon <pus...@kaist.ac.kr> wrote:
>>
>>> Actually, I used preempt_lock to prevent data races.
>>> If two concurrent threads in a core access same per-cpu variable, I
>>> think we still need preempt lock.
>>>
>>
>> This is true - if you have two threads in the same core that access the
>> same per-cpu variable, you need some sort of locking.
>> preempt lock isn't the only way, of course - you can also use a mutex, as
>> well as std::atomic, and other solutions.
>>
>> preempt lock is indeed usually the fastest method, but as you saw it
>> comes with strings attached - the locked code really
>> cannot cause any preemption - which means it can't wait for any mutex,
>> cannot do anything (including delayed symbol
>> resolution which might wait for a mutex). In addition, you need to make
>> sure the entire object is already in memory and
>> doesn't need to be demand-paged, or you may get a preemption in the
>> middle of the code just to read in another page
>> of executable.
>>
>> We have a macro OSV_ELF_MLOCK_OBJECT() (from <osv/elf.hh>) which marks
>> the object with a flag (a
>> .note.osv-mlock section) that ensures *both* things: The object is
>> entirely read into memory on start, and all of
>> its symbols are resolved on start. You can see an example of
>> OSV_ELF_MLOCK_OBJECT() being used in a bunch
>> of tests in tests/. If you use this macro, you don't need to change your
>> code's compilation.
>>
>> example)
>>>
>>> counter's initial value: 0
>>>
>>> CPU 0
>>> Thread A A_local = counter + 1 (A_local = 1)
>>> Thread A *(preemption)*
>>> Thread B B_local = counter + 1 (B_local = 1)
>>> Thread B counter = B_local (counter = 1)
>>> Thread B *(exit)*
>>> Thread A counter = A_local (counter = 1)
>>>
>>> I expect counter to be 2, but 1 returns.
>>>
>>>
>>> 2020년 5월 26일 화요일 오후 3시 4분 24초 UTC+9, Nadav Har'El 님의 말:
>>>>
>>>>
>>>> On Tue, May 26, 2020 at 4:22 AM Wonsup Yoon <pus...@kaist.ac.kr> wrote:
>>>>
>>>>> Thank you for the response.
>>>>>
>>>>> Yes, dynamic_percpu<T> is perfect for my purpose.
>>>>>
>>>>> However, I encountered another issue.
>>>>>
>>>>> If I use dynamic_percpu with preempt-lock (I think it is very common
>>>>> pattern), it abort due to assertion failed.
>>>>> It seems lazy binding prevents preemption lock.
>>>>> So, I had to add -fno-plt option, and it works.
>>>>>
>>>>
>>>> You are right about preempt lock and your workaround for lazy binding.
>>>> However, to use a per-cpu variable, you don't need full preemption
>>>> locking - all you need is *migration* locking - in other words, the thread
>>>> running this code should not be migrated to a different CPU (this will
>>>> change the meaning of the per-cpu variable while you're using it), but it
>>>> is perfectly fine for the thread to be preempted to run a different thread
>>>> - as long as the original thread eventually returns to run on the same CPU
>>>> it previously ran on.
>>>>
>>>> So just replace your use of "preempt_lock" by "migration_lock" (include
>>>> <osv/migration-lock.hh>) and everything should work, without disabling
>>>> lazy
>>>> binding.
>>>>
>>>> Please note that if you use the per-cpu on a thread which is already
>>>> bound to a specific CPU (which was the case in your original code you
>>>> shared), you don't even need migration lock! A pinned thread already can't
>>>> migrate to any other CPU, so it doesn't need to use this
>>>> migration-avoidance mechanism at all. You can use per-cpu variables on
>>>> such
>>>> threads without any special protection.
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>> example code)
>>>>>
>>>>> #include <stdio.h>
>>>>> #include <assert.h>
>>>>>
>>>>> #include <osv/preempt-lock.hh>
>>>>> #include <osv/percpu.hh>
>>>>>
>>>>> struct counter {
>>>>> int x = 0;
>>>>>
>>>>> void inc(){
>>>>> x += 1;
>>>>> }
>>>>>
>>>>> int get(){
>>>>> return x;
>>>>> }
>>>>> };
>>>>>
>>>>> dynamic_percpu<counter> c;
>>>>>
>>>>> int main(int argc, char *argv[])
>>>>> {
>>>>> SCOPE_LOCK(preempt_lock);
>>>>> c->inc();
>>>>>
>>>>> return 0;
>>>>> }
>>>>>
>>>>>
>>>>> Backtrace)
>>>>>
>>>>> [backtrace]
>>>>> 0x000000004023875a <__assert_fail+26>
>>>>> 0x000000004035860c <elf::object::resolve_pltgot(unsigned int)+492>
>>>>> 0x0000000040358669 <elf_resolve_pltgot+57>
>>>>> 0x000000004039e2ef <???+1077535471>
>>>>> 0x000010000000f333 <???+62259>
>>>>> 0x000000004042a47c <osv::application::run_main()+60>
>>>>> 0x0000000040224bd0 <osv::application::main()+144>
>>>>> 0x000000004042a628 <???+1078109736>
>>>>> 0x0000000040462715 <???+1078339349>
>>>>> 0x00000000403fac86 <thread_main_c+38>
>>>>> 0x000000004039f632 <???+1077540402>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2020년 5월 24일 일요일 오후 5시 26분 17초 UTC+9, Nadav Har'El 님의 말:
>>>>>>
>>>>>>
>>>>>> On Sat, May 23, 2020 at 6:35 PM Wonsup Yoon <pus...@kaist.ac.kr>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm trying to use PERCPU macro in application or module.
>>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> The PERCPU macro does not support this. What it does is to add
>>>>>> information about this variable in a special section of the executable
>>>>>> (".percpu"), then arch/x64/loader.ld makes sure all these entries will
>>>>>> be
>>>>>> together between "_percpu_start" and "_percpu_end", and finally sched.cc
>>>>>> for every CPU creates (in the cpu::cpu(id) constructor) a copy of this
>>>>>> data. So if a loadable module (share library) contains another per-cpu
>>>>>> variable, it never gets added to the percpu area.
>>>>>>
>>>>>> However, I believe we do have a mechanism that will suite you:
>>>>>> *dynamic_percpu<T>*.
>>>>>> You can create (and destroy) such an object of type dynamic_percpu<T>
>>>>>> at any time, and it does the right thing: The variable will be
>>>>>> allocated
>>>>>> on all CPUs when the object is created, will be allocated on new cpus if
>>>>>> those happen, and will be freed when the object is destroyed.
>>>>>> In your case you can have a global dynamic_percpu<T> variable in your
>>>>>> loadable module. This object will be created when the module is loaded,
>>>>>> and
>>>>>> destroyed when the module is unloaded - which is what you want.
>>>>>>
>>>>>> Nadav.
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "OSv Development" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to osv...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/osv-dev/07f76c69-0448-4a97-b587-995f7dbafe58%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/osv-dev/07f76c69-0448-4a97-b587-995f7dbafe58%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "OSv Development" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to osv...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/osv-dev/6f9dd47a-4f7a-46a8-89c4-fcaf1909dcc8%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/osv-dev/6f9dd47a-4f7a-46a8-89c4-fcaf1909dcc8%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
--
You received this message because you are subscribed to the Google Groups "OSv
Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to osv-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/osv-dev/6ccc3dd5-9377-4372-9adf-7538c22eee71%40googlegroups.com.
