Jira (PUP-3064) Prevent race condition in Windows service code from prematurely exiting win32-service Service_Main

[Ethan Brown (JIRA)]

Title: Message Title

Ethan Brown commented on an issue   Re: Prevent race condition in Windows service code from prematurely exiting win32-service Service_Main During the last observation of the service code termination, the following threads were observed prior to the SCM stop request, and then were observed after the stop request was initiated, the order tracked as they shut down. Those labelled with TOAST were active threads prior to the STOP request, but no longer running once threads began to exit. Those labelled DEAD were killed in the order given - there are notes about how each thread maps to the Ruby code, and any additional info. Therefore, this can be used as a key in any additional debugging. The important thing to note is that the thread exit order here is not what is expected. code TOAST ef8 ntdll!ZwDelayExecution+0xa KERNELBASE!SleepEx+0xab x64_msvcrt_ruby200!rb_mutex_unlock+0x186 x64_msvcrt_ruby200!ruby_thread_has_gvl_p+0x30b x64_msvcrt_ruby200!rb_vm_localjump_error+0x55ba x64_msvcrt_ruby200!rb_vm_localjump_error+0x5be9 x64_msvcrt_ruby200!rb_funcall+0xd1e x64_msvcrt_ruby200!rb_funcall+0xeb0 x64_msvcrt_ruby200!rb_vm_invoke_proc+0x27 x64_msvcrt_ruby200!rb_thread_lock_destroy+0x7bc x64_msvcrt_ruby200!rb_thread_lock_destroy+0xa6c kernel32!BaseThreadInitThunk+0xd ntdll!RtlUserThreadStart+0x1d TOAST - f3c ntdll!NtWaitForWorkViaWorkerFactory+0xa ntdll!TppWorkerThread+0x2c9 kernel32!BaseThreadInitThunk+0xd ntdll!RtlUserThreadStart+0x1d e7c - DEAD - 1 ntdll!NtWaitForMultipleObjects+0xa KERNELBASE!WaitForMultipleObjectsEx+0xe8 kernel32!WaitForMultipleObjects+0xb0 x64_msvcrt_ruby200!rb_mutex_trylock+0x125 x64_msvcrt_ruby200!rb_mutex_trylock+0x3b0 x64_msvcrt_ruby200!rb_thread_kill+0x30f x64_msvcrt_ruby200!rb_last_status_get+0xd3a x64_msvcrt_ruby200!rb_error_arity+0x125 x64_msvcrt_ruby200!rb_f_send+0x5a1 x64_msvcrt_ruby200!rb_vm_localjump_error+0x1cf8 x64_msvcrt_ruby200!rb_vm_localjump_error+0x5be9 x64_msvcrt_ruby200!rb_funcall+0xd1e x64_msvcrt_ruby200!rb_funcall+0xeb0 x64_msvcrt_ruby200!rb_vm_invoke_proc+0x27 x64_msvcrt_ruby200!rb_thread_lock_destroy+0x7bc x64_msvcrt_ruby200!rb_thread_lock_destroy+0xa6c kernel32!BaseThreadInitThunk+0xd ntdll!RtlUserThreadStart+0x1d 94c - DEAD - 2 ntdll!NtWaitForSingleObject+0xa KERNELBASE!WaitForSingleObjectEx+0x79 ffi_c!Init_ffi_c+0x51b4 x64_msvcrt_ruby200!rb_thread_polling+0x103 ffi_c!Init_ffi_c+0x502a x64_msvcrt_ruby200!rb_thread_lock_destroy+0x60a x64_msvcrt_ruby200!rb_thread_lock_destroy+0xa6c kernel32!BaseThreadInitThunk+0xd ntdll!RtlUserThreadStart+0x1d edc – ThreadProc, which should be native thread based on using CreateThread to create it - DEAD - 3 ntdll!NtWaitForSingleObject+0xa KERNELBASE!WaitForSingleObjectEx+0x79 sechost!ScSendResponseReceiveControls+0x13b sechost!ScDispatcherLoop+0x121 sechost!StartServiceCtrlDispatcherA+0x6d ffi_c!Init_ffi_c+0x90b7 0x2baea10 0x563c60 850 - DEAD - 4 ntdll!NtWaitForSingleObject+0xa KERNELBASE!WaitForSingleObjectEx+0x79 x64_msvcrt_ruby200!rb_thread_list+0x11af kernel32!BaseThreadInitThunk+0xd ntdll!RtlUserThreadStart+0x1d ea0 - thr object busyloop inside of Ruby thread in mainloop - DEAD - 5 ntdll!NtWaitForSingleObject+0xa KERNELBASE!WaitForSingleObjectEx+0x79 ffi_c!Init_ffi_c+0x55ac ffi_c!Init_ffi_c+0x8b7a ffi_c!Init_ffi_c+0x900f 0x430000 0x31aff30 8cc - DEAD - 6 ntdll!NtWaitForWorkViaWorkerFactory+0xa ntdll!TppWorkerThread+0x2c9 kernel32!BaseThreadInitThunk+0xd ntdll!RtlUserThreadStart+0x1d ee4 – mainloop, owner of events, and thread handle - DEAD - 7 ntdll!NtWaitForSingleObject+0xa KERNELBASE!WaitForSingleObjectEx+0x79 ffi_c!Init_ffi_c+0x55ac ffi_c!Init_ffi_c+0x8b7a ffi_c!Init_ffi_c+0x900f 0x440000 0x279ff60 9fc – line 215 of Service_Main – part that ends up dying prematurely... DEAD - 8 ntdll!NtWaitForMultipleObjects+0xa KERNELBASE!WaitForMultipleObjectsEx+0xe8 kernel32!WaitForMultipleObjects+0xb0 x64_msvcrt_ruby200!rb_mutex_trylock+0x125 x64_msvcrt_ruby200!rb_mutex_trylock+0x3b0 x64_msvcrt_ruby200!rb_thread_kill+0x30f x64_msvcrt_ruby200!rb_last_status_get+0xd3a x64_msvcrt_ruby200!rb_error_arity+0x125 x64_msvcrt_ruby200!rb_vm_localjump_error+0x1cf8 x64_msvcrt_ruby200!rb_vm_localjump_error+0x5be9 x64_msvcrt_ruby200!rb_funcall+0xd1e x64_msvcrt_ruby200!rb_funcall+0xeb0 x64_msvcrt_ruby200!rb_vm_invoke_proc+0x27 x64_msvcrt_ruby200!rb_block_lambda+0x7b3 x64_msvcrt_ruby200!rb_vm_localjump_error+0x6147 x64_msvcrt_ruby200!rb_funcall2+0x8f ffi_c!Init_ffi_c+0x5ccb x64_msvcrt_ruby200!rb_rescue2+0x18a ffi_c!Init_ffi_c+0x548f x64_msvcrt_ruby200!rb_thread_lock_destroy+0x60a 5dc ntdll!NtWaitForMultipleObjects+0xa KERNELBASE!WaitForMultipleObjectsEx+0xe8 kernel32!WaitForMultipleObjects+0xb0 x64_msvcrt_ruby200!rb_mutex_trylock+0x125 x64_msvcrt_ruby200!rb_mutex_trylock+0x522 x64_msvcrt_ruby200!rb_thread_sleep_forever+0x24e x64_msvcrt_ruby200!rb_thread_sleep_forever+0x44a x64_msvcrt_ruby200!rb_ensure+0xda x64_msvcrt_ruby200!rb_thread_list+0x1019 x64_msvcrt_ruby200!rb_thread_list+0x1157 x64_msvcrt_ruby200!rb_error_arity+0x125 x64_msvcrt_ruby200!rb_f_send+0x5a1 x64_msvcrt_ruby200!rb_vm_localjump_error+0x1cf8 x64_msvcrt_ruby200!rb_vm_localjump_error+0x5be9 x64_msvcrt_ruby200!rb_iseq_eval_main+0x1e0 x64_msvcrt_ruby200!rb_check_copyable+0x3266 x64_msvcrt_ruby200!ruby_run_node+0x51 ruby!main+0x54 ruby!__tmainCRTStartup+0x257 ruby!mainCRTStartup+0x18 code Add Comment   Puppet / PUP-3064 Prevent race condition in Windows service code from prematurely exiting win32-service Service_Main A workaround has been put in place as part of PUP-2987 to fix issues discovered in the win32-service gem. It also has slightly restructured the Windows service daemon. This should resolve deadlocks experienced when terminating the service. However, the reason that this works is due to the fact that {{Service_Main}} now contains an ensure block that... This message was sent by Atlassian JIRA (v6.1.4#6159-sha1:44eaede) -- You received this message because you are subscribed to the Google Groups "Puppet Bugs" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-bugs+unsubscr...@googlegroups.com. To post to this group, send email to puppet-bugs@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-bugs. For more options, visit https://groups.google.com/d/optout.