It's possible there's a bug. It's also possible you're getting really unlucky and the uninitialized mutex looks valid because the magic is valid.
On Sun, Apr 21, 2019, 06:10 Rouven WEILER <[email protected]> wrote: > Dear all, > > trying to make samba work and getting rid of my error message > > [2019/04/01 09:51:41.112485, 0] > ../../lib/tdb_wrap/tdb_wrap.c:64(tdb_wrap_log) > tdb(/var/samba/lock/gencache.tdb): tdb_open_ex: tdb_mutex_init failed for > /var/samba/lock/gencache.tdb: Device busy > > > I followed the info of Youzhong. It definitely works. So thankx. > BUT: The OS (OpenIndiana; state of April,19th) is patched against the > illumos issue 9959 > > So there might be some mechanism (maybe MUTEX_MAGIC) that avoids the > initialization of the mutex before init. > I am really not deep in all that pthread stuff… > > Is there the possiblity that there is a bug in the code base of issue 9959? > > > Best regards > > Rouven > > > Am 18.04.2019 um 19:20 schrieb Robert Mustacchi <[email protected]>: > > On 4/18/19 10:12 , Youzhong Yang wrote: > > Little bit confused. The page https://www.openindiana.org/download/ shows > latest ISO is dated 2018.10, but 9959 fix was submitted on 11/26/2018, I > guess Hipster doesn't have it. > > > The ISO is just the install image. Generally, after using that to > bootstrap, then folks will update to latest bits via IPS. As a result, > the ISO isn't updated (and generally doesn't have to be) quite as often. > > Robert > > On Thu, Apr 18, 2019 at 12:38 PM Rouven WEILER <[email protected]> > wrote: > > On my system the issue is solved and the sources are patched accordingly. > Thus, the issue seems to be somewhere else. > > Am 18.04.2019 um 18:27 schrieb Rouven WEILER <[email protected]>: > > concerning openindiana I am using the most recent one. I right now asked > if the issue is solved in the latest version. > > another interesting thing is that older samba versions do not throw EBUSY, > although on the same underlying system. > > > > Am 18.04.2019 um 18:07 schrieb Youzhong Yang <[email protected]>: > > That is interesting. I am curious about how pthread_mutexattr_init ends up > returning EBUSY. What verison of Openindiana are you running? is there a > link pointing to the fix that it has for 9959? > > On Thu, Apr 18, 2019 at 11:53 AM Rouven WEILER <[email protected]> > wrote: > > I understand and see that it works, but as written below > pthread_mutexattr_init throws EBUSY, not pthread_mutex_init. > > Also Openindiana seems to include the patch for issue 9959 mentioned. I > will clarify that to make sure. > > I just want to make sure that the patch you suggest aims for the issue > and is not a workaround. > > > Am 18.04.2019 um 16:09 schrieb Youzhong Yang <[email protected]>: > > I reproduced the issue on a smartos version which does not have the fix > for https://www.illumos.org/issues/9959. > > The pthread_mutex_init() at line 590 returns EBUSY: > ret = pthread_mutex_init(chain, &ma); > > > https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l590 > > > Then I applied the following patch to lib/tdb/common/mutex.c: > > # git grep -B 1 -n pthread_mutex_init lib/tdb/common/mutex.c > lib/tdb/common/mutex.c-590- bzero(chain, > sizeof(pthread_mutex_t)); > lib/tdb/common/mutex.c:591: ret = pthread_mutex_init(chain, > &ma); > -- > lib/tdb/common/mutex.c-599- bzero(&m->allrecord_mutex, > sizeof(pthread_mutex_t)); > lib/tdb/common/mutex.c:600: ret = > pthread_mutex_init(&m->allrecord_mutex, &ma); > -- > lib/tdb/common/mutex.c-693- bzero(&m, sizeof(pthread_mutex_t)); > lib/tdb/common/mutex.c:694: ret = pthread_mutex_init(&m, &ma); > -- > lib/tdb/common/mutex.c-891- bzero(ptr, sizeof(pthread_mutex_t)); > lib/tdb/common/mutex.c:892: ret = pthread_mutex_init(ptr, &ma); > > It works like a charm. Fair enough? > > So again I recommend either of the following two: > - apply 9959 fix to your OS image, or use a newer version which has 9959 > fix > - bzero pthread_mutex_t struct before calling pthread_mutex_init(). > > In my environment I applied bzero patch to all pthread_mutex_init() calls > in the entire samba code base, you know, who knows what is going to happen. > > > > On Thu, Apr 18, 2019 at 4:29 AM Rouven WEILER <[email protected]> > wrote: > > That sounds reasonable! > Could you help me to dtrace that in a feasible way? Up to now I am not > used to dtrace. Except knowing what is and that its powerful. > > Am 18.04.2019 um 09:29 schrieb Jason King <[email protected]>: > > It appears that the robust mutexes are mmapped from the .tdb file. > EBUSY means that the mutex has already been intialized. It also appears > that if the TDB_CLEAR_IF_FIRST flag is _not_ set when opening the tdb, it > will try to initialize the mutexes. I’m wondering if the tdb file gets > created, the memory image of the initialized mutexes is getting saved, then > when it’s opened again, if that flag is not set, it tries to re-initialize > the mutexes and fails (because they are already constructed) legitimately > returning EBUSY. If the samba code is in fact trying to reinitialize an > already initialized mutex (dtrace might be able to help determine this — at > this point it’s merely a supposition), that would explain the error. > > > > > From: Rouven WEILER <[email protected]> <[email protected]> > Reply: illumos-developer <[email protected]> > <[email protected]> > Date: April 18, 2019 at 12:48:48 AM > To: Youzhong Yang <[email protected]> <[email protected]> > Cc: Robert Mustacchi <[email protected]> <[email protected]>, Toomas Soome via > illumos-developer <[email protected]> > <[email protected]> > Subject: [developer] Aw: Re: [Samba] gencache.tdb: device busy > > You are totally right. That's the same I got from the man pages. > > The responsible code lines of samba also seems to bear no issue and also > this issue arises on samba version 4.10.x not on earlier ones. > In addition, there seems to be only a problem with > illumos/openindiana... > That's why I do not know how to get that fixed. Up to now I hadn't time > to dtrace. > > Only thing I got so far: When the mutexattr is already set the return > value will result on undefined behaviour. > But why only after version 4.10.x on openindiana/illumos? > > Just for completion here the codelines of samba: > > https://git.samba.org/?p=samba.git;a=blob;f=lib/tdb/common/mutex.c;h=8a122d5d6a2ec321bce25cf06062acc808f1e126;hb=HEAD#l570 > > > > Gesendet: Mittwoch, 17. April 2019 um 19:19 Uhr > Von: "Youzhong Yang" <[email protected]> > An: "Rouven WEILER" <[email protected]> > Cc: "Robert Mustacchi" <[email protected]>, "Toomas Soome via > illumos-developer" <[email protected]> > Betreff: Re: [Samba] gencache.tdb: device busy > > Unless I am totally wrong, pthread_mutexattr_init() shouldn't return > EBUSY: > > > http://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libc/port/threads/pthr_mutex.c#37 > > > On Tue, Apr 16, 2019 at 2:45 PM Rouven WEILER <[email protected][ > mailto:[email protected] <[email protected]>]> wrote:Hi Robert > & Youzhong, > > I got a response from the Samba-Devs. > Would the issue mentioned by Youzhong also hot pthread_mutexattr_t? > > Concerning dtrace I hadn’t the time yet but I will check for that. > Is there maybe a way to trace all forks of smbd? Usually the issue > arises after the fork… > > FIND BELOW THE STATEMENT FROM samba.org[http://samba.org]: > > This doesn't look exactly the same. In your > log below you say: > > ret = pthread_mutexattr_init(&ma); in line 570 > I got errorno 16 --> device busy. > > But the bug report is about needing the value > passed to pthread_mutex_init() needing to be > zero, not the value passed to pthread_mutexattr_init(). > > Can you just try adding '= {0}' to all the > places where we declare: > > pthread_mutexattr_t ma > > (i.e. make it say) > > pthread_mutexattr_t ma = {0}; > > and see if it makes your problem go away ? > > END OF STATEMENT > > > Anfang der weitergeleiteten Nachricht: > > Von: Jeremy Allison via samba <[email protected][ > > mailto:[email protected] <[email protected]>]> > > Betreff: Aw: [Samba] gencache.tdb: device busy > Datum: 16. April 2019 um 02:07:05 MESZ > An: Rouven WEILER <[email protected][mailto:[email protected] > > <[email protected]>]> > > Kopie: [email protected][mailto:[email protected] > > <[email protected]>] > > Antwort an: Jeremy Allison <[email protected][mailto:[email protected] > > <[email protected]>]> > > > On Fri, Apr 12, 2019 at 09:46:04AM +0200, Rouven WEILER via samba > > wrote: > > > > Hi Jeremy, > > I got some info on that topic from the illumos devs: > > > It's a sporadic issue, you're lucky enough to not encounter it on > > 4.9.5. > > > I confirmed in 4.10.2, it happens: > > winbindd.log: tdb(/tmw-nas-3p/samba/var/lock/gencache.tdb): > > tdb_open_ex: tdb_mutex_init failed for > /tmw-nas-3p/samba/var/lock/gencache.tdb: Device busy > > > So either apply OS fix, or apply a software fix (zeroing > > pthread_mutex_t before pthread_mutex_init()). > > > This all should be based on the following issue: > > > https://www.illumos.org/issues/9959[https://www.illumos.org/issues/9959] > > > As far as I understand the issue making an software fix in samba for > > the OS seems more reasonable since the problem can occur sporadic and thus > again... > > Could you tell me how to help to fix that bug? > > > This doesn't look exactly the same. In your > log below you say: > > ret = pthread_mutexattr_init(&ma); in line 570 > I got errorno 16 --> device busy. > > But the bug report is about needing the value > passed to pthread_mutex_init() needing to be > zero, not the value passed to pthread_mutexattr_init(). > > Can you just try adding '= {0}' to all the > places where we declare: > > pthread_mutexattr_t ma > > (i.e. make it say) > > pthread_mutexattr_t ma = {0}; > > and see if it makes your problem go away ? > > > Gesendet: Dienstag, 02. April 2019 um 18:59 Uhr > Von: "Jeremy Allison" <[email protected][mailto:[email protected] > > <[email protected]>]> > > An: "Rouven WEILER" <[email protected][ > > mailto:[email protected] <[email protected]>]> > > Cc: [email protected][mailto:[email protected] > > <[email protected]>] > > Betreff: Re: Re: [Samba] gencache.tdb: device busy > On Tue, Apr 02, 2019 at 01:24:19PM +0200, Rouven WEILER wrote: > > > > I added LOG statements to mutex.c printing the strerror from the > > return values. > > > From > > ret = pthread_mutexattr_init(&ma); in line 570 > > I got errorno 16 --> device busy. > > But due the manual of pthread_mutex_attr_init() there should only be > > ENOMEM and EINVAL... > > > Doing gdb smbd -> run > Doe not reveal anything, but I never used gdb for debugging > > anything. I just gave it a shot... > > > Sorry, looks a little like the libc/pthread_mutex support > on Solaris is broken. You might want to raise this on a > Solaris/Illumos forum. > > -- > To unsubscribe from this list go to the following URL and read the > instructions: > > > https://lists.samba.org/mailman/options/samba[https://lists.samba.org/mailman/options/samba] > > > -- > To unsubscribe from this list go to the following URL and read the > instructions: > > > https://lists.samba.org/mailman/options/samba[https://lists.samba.org/mailman/options/samba] > > > ------------------------------------------ > illumos: illumos-developer > Permalink: > > https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M99a2afeb19714648c051ab5f > Delivery options: > https://illumos.topicbox.com/groups/developer/subscription > > *illumos <https://illumos.topicbox.com/latest>* / illumos-developer / > > see discussions <https://illumos.topicbox.com/groups/developer> + > participants <https://illumos.topicbox.com/groups/developer/members> + > delivery > options <https://illumos.topicbox.com/groups/developer/subscription> > Permalink > < > https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M2372380a78d4d5078f031f0c > > > > > > ------------------------------------------ > illumos: illumos-developer > Permalink: > https://illumos.topicbox.com/groups/developer/T38b7c9b3f92fa699-M1048b12f29cda459a4de0893 > Delivery options: > https://illumos.topicbox.com/groups/developer/subscription > > > _______________________________________________ > oi-dev mailing list > [email protected] > https://openindiana.org/mailman/listinfo/oi-dev >
_______________________________________________ oi-dev mailing list [email protected] https://openindiana.org/mailman/listinfo/oi-dev
