I’ve started getting a consistent hang in one of our unit tests, in which we’re
uploading 30MB of data to an HTTP server via a PUT request. The body is a
stream of data generated at runtime; to stream this to an NSURLTask we call
CFStreamCreateBoundPair, set the resulting input stream as the task’s
HTTPBodyStream, and then schedule the paired output stream with our runloop and
write to it when notified space is available.
This is on OS X 10.11.4 (with both of the latest two betas.) It hasn’t occurred
on another machine running 10.11.3. I’m going to file a Radar, but I’m also
asking here in case anyone else has run into this and/or knows of a workaround.
My code's write call blocks forever like this — it’s been notified the output
stream has space, but while writing the output stream is stuck acquiring a
mutex:
frame #0: 0x00007fff88e8cde6 libsystem_kernel.dylib`__psynch_mutexwait + 10
frame #1: 0x00007fff965b7e4a
libsystem_pthread.dylib`_pthread_mutex_lock_wait + 89
frame #2: 0x00007fff9528e274 CoreFoundation`boundPairWrite + 52
frame #3: 0x00007fff951aafa5 CoreFoundation`CFWriteStreamWrite + 437
* frame #4: 0x0000000107116b5d CouchbaseLite`-[CBLMultiStreamWriter
writeToOutput](self=0x000060c00004f780, _cmd="writeToOutput") + 429 at
CBLMultiStreamWriter.m:254
frame #5: 0x0000000107117712 CouchbaseLite`-[CBLMultiStreamWriter
stream:handleEvent:](self=0x000060c00004f780, _cmd="stream:handleEvent:",
stream=0x000060d000056170, event=NSStreamEventHasSpaceAvailable) + 1506 at
CBLMultiStreamWriter.m:288
frame #6: 0x00007fff95200094 CoreFoundation`_signalEventSync + 180
frame #7: 0x00007fff9521d29e
CoreFoundation`_cfstream_shared_signalEventSync + 478
frame #8: 0x00007fff951de881
CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
At the same time, the CFNetwork thread is also blocked inside boundPairRead,
presumably the mate of the stream we’re writing to:
* thread #7: tid = 0x5393e, 0x00007fff88e8cde6
libsystem_kernel.dylib`__psynch_mutexwait + 10, name =
'com.apple.NSURLConnectionLoader'
* frame #0: 0x00007fff88e8cde6 libsystem_kernel.dylib`__psynch_mutexwait + 10
frame #1: 0x00007fff965b7e4a
libsystem_pthread.dylib`_pthread_mutex_lock_wait + 89
frame #2: 0x00007fff9528dc57 CoreFoundation`boundPairRead + 151
frame #3: 0x00007fff951b49c5 CoreFoundation`CFReadStreamRead + 389
frame #4: 0x00007fff96c22b1e
CFNetwork`RequestBodyStreamProvider::readBodyStream(bool) + 308
frame #5: 0x00007fff95200094 CoreFoundation`_signalEventSync + 180
frame #6: 0x00007fff9521c67b CoreFoundation`_cfstream_solo_signalEventSync
+ 251
frame #7: 0x00007fff951fd8fc CoreFoundation`_CFStreamSignalEvent + 476
frame #8: 0x00007fff9528dec8 CoreFoundation`boundPairRead + 776
frame #9: 0x00007fff951b49c5 CoreFoundation`CFReadStreamRead + 389
frame #10: 0x00007fff96c22b1e
CFNetwork`RequestBodyStreamProvider::readBodyStream(bool) + 308
frame #11: 0x00007fff95200094 CoreFoundation`_signalEventSync + 180
frame #12: 0x00007fff9521d29e
CoreFoundation`_cfstream_shared_signalEventSync + 478
frame #13: 0x00007fff951de881
CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE0_PERFORM_FUNCTION__ + 17
frame #14: 0x00007fff951bdf37 CoreFoundation`__CFRunLoopDoSources0 + 423
frame #15: 0x00007fff951bd4df CoreFoundation`__CFRunLoopRun + 927
frame #16: 0x00007fff951bced8 CoreFoundation`CFRunLoopRunSpecific + 296
frame #17: 0x00007fff96a71111 CFNetwork`+[NSURLConnection(Loader)
_resourceLoadLoop:] + 412
The latter stack looks very suspicious — it’s got two re-entrant calls to
RequestBodyStreamProvider::readBodyStream stacked on top of each other. Frame
12 is a stream signaling an event, which triggers a read in frame 9, but
somehow the read triggers another event in frame 6, which triggers a nested
read in frame 3.
This kind of re-entrant call usually smells trouble to me, and in this case I
strongly suspect that the outer call has a lock on the stream’s mutex, and so
when the inner call tries to lock it again it deadlocks. This would then cause
my thread to get stuck to the tar-baby as well when it tries to write to the
paired stream.
—Jens _______________________________________________
Do not post admin requests to the list. They will be ignored.
Macnetworkprog mailing list (Macnetworkprog@lists.apple.com)
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/macnetworkprog/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
