Re: [Mingw-w64-public] Deadlock in winpthreads' pthread_cond_signal()

[Sitsofe Wheeler]

On 10 March 2018 at 12:55, Liu Hao  wrote:
> 在 2018/3/10 20:46, Sitsofe Wheeler 写道:
>> On 10 March 2018 at 12:38, Liu Hao  wrote:
>>> 在 2018/3/10 20:06, Sitsofe Wheeler 写道:
 Reattaching example program with .txt extension and just in case you
 can also download it from http://sucs.org/~sits/test/signalhang.c .

>>>
>>> This seems not producible on my Windows 7, but it is indeed reproducible
>>> on Wine 3.3 on Linux Mint 18.3. This renders the bug valid,
>>
>> It definitely happened on 2012 R2 for me (the backtrace I gave was
>> from running on 2012 R2) but I believe it was more intermittent than
>> under Wine. I suggest raising thread_count to 100 and seeing if that
>> makes it more reproducible on native Windows.
>
> No, not reproducible at all, even with 1000 threads.
>
> I created a ticket for it:
> .
>
>>
>>> I had a peek at 'cond.c' but was still unclear about what was going on.
>>> Further investigation is required.

Find an updated signalhang.c attached as a .txt (also downloadable
from 
https://gist.github.com/sitsofe/abd99efec507b18234e7cc042d1baff1#file-signalhang-c
). This version makes the deadlock very reproducible for me on a 2012
R2 Azure VM with 2 CPUs. Note that one caveat I've found is that if
all the threads are bound to the same CPU the deadlock becomes very
hard to hit to hit on both Wine and the 2012 VM - did your Windows 7
machine have more than one CPU?

-- 
Sitsofe | http://sucs.org/~sits/
/* Program that demonstrates a hang in pthread_cond_signal() when using
 * winpthreads (at least versions 5.0.2-7.2.0). On Windows compile with
 * x86_64-w64-mingw32-gcc -g -O3 -static -pthread signalhang.c -o signalhang
 * on Linux compile with
 * gcc -g -O3 -lrt -pthread signalhang.c -o signalhang
 *
 * Extra compilation options:
 * -DDETECT_DEADLOCK=1 : Print a message and trigger an assert if it looks like
 *   threads have become deadlocked
 * -DLOGGING=1 : Print logging information when wakes are done. "hits" is
 *   incremented when a wake is done and there were two more other
 *   threads waiting on the condition variable.
 * -DNDEBUG : Disable all return code checking asserts and don't trigger an
 *assert after the deadlock is detected when using
 *DETECT_DEADLOCK=1
 *
 * Usage:
 * ./signalhang [THREAD_COUNT]
 * Program will print a done line with some stats on success or hang on
 * deadlock. It is easy to loop runs until a failure (especially when using
 * DETECT_DEADLOCK=1) by using the shell. For example:
 * PS C:\> $LASTEXITCODE=0; While ($LASTEXITCODE -eq 0) { .\signalhang.exe 4 }
 * or
 * $ rc=0; while [[ rc -eq 0 ]]; do ./signalhang 4; rc=$?; done
 *
 * Deadlock does not occur when any of the following are true:
 * - Running on native Linux with glibc
 * - When the call to pthread_cond_signal() is moved inside the
 *   pthread_mutex_lock()/pthread_mutex_unlock() region
 * The following make deadlock less likely:
 * - Binding threads to the same single CPU
 * - Using less than 4 threads
 * The following make deadlock more likely:
 * - More threads
 * - Turning logging on
 * - Calling sched_yield() between mutex_down()/mutex_up()
 * - Binding threads to different CPUs
 */

#ifndef LOGGING
#define LOGGING 0 // setting to 1 increases deadlock probability
#endif
#ifndef DETECT_DEADLOCK
#define DETECT_DEADLOCK 0 // setting to 1 enables self deadlock detection
#endif
#define MAX_MISSED_HEARTBEATS 10 // the max missed heartbeats with deadlock 
detection

#ifdef _WIN32
#include 
#else
#define _GNU_SOURCE
#endif
#include 
#include 
#include 
#include 
#include 
#include 
#include 

#define CHECK(x) if ((x) != 0) assert(0)

#if LOGGING
#define log(x, ...) fprintf(stderr, x, ##__VA_ARGS__)
#else
#define log(x, ...) do {} while (0)
#endif

enum {
MUTEX_LOCKED = 0,
MUTEX_UNLOCKED = 1
};

struct sh_mutex {
uint64_t max_iterations;
uint64_t max_hits;
uint64_t iteration;
uint64_t hits;
int value;
int waiters;
pthread_mutex_t lock;
pthread_cond_t cond;
};

struct sh_thread {
struct sh_mutex *mutex;
int id;
int *alive;
unsigned int *no_heartbeat;
};

static void mutex_down(struct sh_mutex *mutex) {
CHECK(pthread_mutex_lock(>lock));
while (mutex->value == MUTEX_LOCKED) {
mutex->waiters++;
CHECK(pthread_cond_wait(>cond, >lock));
mutex->waiters--;
}
mutex->value = MUTEX_LOCKED;
CHECK(pthread_mutex_unlock(>lock));
}

static void mutex_up(struct sh_mutex *mutex) {
int do_wake = 0;
uint64_t iteration;
uint64_t hits;

CHECK(pthread_mutex_lock(>lock));
if (mutex->value == MUTEX_LOCKED && mutex->waiters)
do_wake = mutex->waiters;
mutex->value = MUTEX_UNLOCKED;
iteration = 

Re: [Mingw-w64-public] Deadlock in winpthreads' pthread_cond_signal()

[Liu Hao]

在 2018/3/10 20:46, Sitsofe Wheeler 写道:
> On 10 March 2018 at 12:38, Liu Hao  wrote:
>> 在 2018/3/10 20:06, Sitsofe Wheeler 写道:
>>> Reattaching example program with .txt extension and just in case you
>>> can also download it from http://sucs.org/~sits/test/signalhang.c .
>>>
>>
>> This seems not producible on my Windows 7, but it is indeed reproducible
>> on Wine 3.3 on Linux Mint 18.3. This renders the bug valid,
> 
> It definitely happened on 2012 R2 for me (the backtrace I gave was
> from running on 2012 R2) but I believe it was more intermittent than
> under Wine. I suggest raising thread_count to 100 and seeing if that
> makes it more reproducible on native Windows.

No, not reproducible at all, even with 1000 threads.

I created a ticket for it:
.

> 
>> I had a peek at 'cond.c' but was still unclear about what was going on.
>> Further investigation is required.
> 


-- 
Best regards,
LH_Mouse
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Mingw-w64-public mailing list
Mingw-w64-public@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mingw-w64-public

Re: [Mingw-w64-public] Deadlock in winpthreads' pthread_cond_signal()

[Sitsofe Wheeler]

On 10 March 2018 at 12:38, Liu Hao  wrote:
> 在 2018/3/10 20:06, Sitsofe Wheeler 写道:
>> Reattaching example program with .txt extension and just in case you
>> can also download it from http://sucs.org/~sits/test/signalhang.c .
>>
>
> This seems not producible on my Windows 7, but it is indeed reproducible
> on Wine 3.3 on Linux Mint 18.3. This renders the bug valid,

It definitely happened on 2012 R2 for me (the backtrace I gave was
from running on 2012 R2) but I believe it was more intermittent than
under Wine. I suggest raising thread_count to 100 and seeing if that
makes it more reproducible on native Windows.

> I had a peek at 'cond.c' but was still unclear about what was going on.
> Further investigation is required.

-- 
Sitsofe | http://sucs.org/~sits/

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Mingw-w64-public mailing list
Mingw-w64-public@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mingw-w64-public

Re: [Mingw-w64-public] Deadlock in winpthreads' pthread_cond_signal()

[Sitsofe Wheeler]

On 10 March 2018 at 10:37, Liu Hao  wrote:
> 在 2018/3/10 18:14, Sitsofe Wheeler 写道:
>> Hi,
>>
>> It's possible to trigger a deadlock in winpthreads'
>> pthread_cond_signal() when using the pthread_cond_signal() call
>> outside of a pthread_mutex_lock()/pthread_mutex_unlock() region. This
>> problem happens under Windows 2012 R2 with winpthreads-5.0.3
>> (Cygwin64) and winpthreads-7.2.0 (standalone) and also on Fedora 26
>> under wine-3.2-2.fc26.x86_64 using
>> mingw64-winpthreads-static-5.0.2-1.fc26.noarch. The problem was
>> originally seen by a Windows user of the fio project (see
>> https://www.spinics.net/lists/fio/msg06772.html ) but a standalone
>> program that reproduces the problem is attached. The problem is
>
> The attachment seemed swallowed by SourceForge. If it was a plaintext
> file, add a `.txt` extension and send it again.
>
> Thanks for your report.
>
>> intermittent (so you may need to up the threads/iterations or run the
>> program multiple times) but a backtrace of the thread with the hang
>> looks like this:

Reattaching example program with .txt extension and just in case you
can also download it from http://sucs.org/~sits/test/signalhang.c .

-- 
Sitsofe | http://sucs.org/~sits/
/* Program that demonstrates a hang in pthread_cond_signal when using
 * winpthreads. Compile with
 * x86_64-w64-mingw32-gcc -O3 -Wall -static -pthread signal_hang.c
 */

#include 
#include 
#include 
#include 
#include 
#include 

#define CHECK(x, msg) \
if (x != 0) { \
printf("%s failed\n", msg); \
goto err; \
}

enum {
MUTEX_LOCKED = 0,
MUTEX_UNLOCKED = 1
};

struct sh_mutex {
pthread_mutex_t lock;
pthread_cond_t cond;
int value;
int waiters;
uint64_t max_iterations;
uint64_t iteration;
};

struct sh_thread {
int id;
struct sh_mutex *mutex;
};

static void mutex_down(struct sh_mutex *mutex) {
pthread_mutex_lock(>lock);
while (mutex->value == MUTEX_LOCKED) {
mutex->waiters++;
pthread_cond_wait(>cond, >lock);
mutex->waiters--;
}
mutex->value = MUTEX_LOCKED;
pthread_mutex_unlock(>lock);
}

static void mutex_up(struct sh_mutex *mutex) {
int do_wake = 0;
uint64_t iteration;

pthread_mutex_lock(>lock);
if (mutex->value == MUTEX_LOCKED && mutex->waiters)
do_wake = mutex->waiters;
mutex->value = MUTEX_UNLOCKED;
iteration = mutex->iteration;
pthread_mutex_unlock(>lock);

if (do_wake) {
fprintf(stderr, "doing wake (waiters=%d, iteration=%" PRId64 
")...\n", do_wake, iteration);
pthread_cond_signal(>cond);
}
}

static void *contend_lock_thread(void *data) {
struct sh_thread *thread_data = (struct sh_thread*) data;
struct sh_mutex *mutex = thread_data->mutex;

while (1) {
uint64_t iteration;

mutex_down(mutex);
mutex->iteration++;
iteration = mutex->iteration;
usleep(5);
mutex_up(mutex);
if (iteration >= mutex->max_iterations)
break;
}

fprintf(stderr, "finishing thread %d\n", thread_data->id);
return NULL;
}

int main(void) {
struct sh_mutex mutex;
struct sh_thread *threads_data = NULL;
pthread_t *threads = NULL;
int thread_count;

CHECK(pthread_cond_init(, NULL), "pthread_cond_init");
CHECK(pthread_mutex_init(, NULL), "pthread_mutex_init");
mutex.value = MUTEX_UNLOCKED;
mutex.iteration = 0;
mutex.waiters = 0;

/* Change these to make the test longer */
thread_count = 50;
mutex.max_iterations = 1;

threads = malloc(sizeof(pthread_t) * thread_count);
threads_data = malloc(sizeof(struct sh_thread) * thread_count);
for (int i = 0; i < thread_count; i++) {
threads_data[i].id = i;
threads_data[i].mutex = 
CHECK(pthread_create([i], NULL, contend_lock_thread, 
_data[i]), "pthread_create");
}

for (int i = 0; i < thread_count; i++) {
CHECK(pthread_join(threads[i], NULL), "pthread_join");
}

if (threads)
free(threads);
if (threads_data)
free(threads_data);

printf("iterations done: %" PRId64 "\n", mutex.iteration);
return 0;
err:
perror("last error was");
return 1;
}
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
Mingw-w64-public mailing list
Mingw-w64-public@lists.sourceforge.net

Re: [Mingw-w64-public] Deadlock in winpthreads' pthread_cond_signal()

[Liu Hao]

在 2018/3/10 18:14, Sitsofe Wheeler 写道:
> Hi,
> 
> It's possible to trigger a deadlock in winpthreads'
> pthread_cond_signal() when using the pthread_cond_signal() call
> outside of a pthread_mutex_lock()/pthread_mutex_unlock() region. This
> problem happens under Windows 2012 R2 with winpthreads-5.0.3
> (Cygwin64) and winpthreads-7.2.0 (standalone) and also on Fedora 26
> under wine-3.2-2.fc26.x86_64 using
> mingw64-winpthreads-static-5.0.2-1.fc26.noarch. The problem was
> originally seen by a Windows user of the fio project (see
> https://www.spinics.net/lists/fio/msg06772.html ) but a standalone
> program that reproduces the problem is attached. The problem is

The attachment seemed swallowed by SourceForge. If it was a plaintext
file, add a `.txt` extension and send it again.

Thanks for your report.

> intermittent (so you may need to up the threads/iterations or run the
> program multiple times) but a backtrace of the thread with the hang
> looks like this:
> 


-- 
Best regards,
LH_Mouse
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
Mingw-w64-public mailing list
Mingw-w64-public@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/mingw-w64-public