zeroshade commented on code in PR #1609:
URL: https://github.com/apache/arrow-go/pull/1609#discussion_r4618064941


##########
arrow/cdata/cdata_exports.go:
##########
@@ -511,17 +512,24 @@ func (rr *cRecordReader) release() {
 }
 
 type cAsyncStreamHandler struct {
+       // mu guards producer: once release returns the producer may be freed,
+       // so it must not be called again.
+       mu        sync.Mutex
        producer  *CArrowAsyncProducer
        taskQueue chan taskState
        ctx       context.Context
 }
 
-func asyncTaskQueue(ctx context.Context, schema *arrow.Schema, recordStream 
chan<- RecordMessage, taskQueue <-chan taskState, producer 
*CArrowAsyncProducer) {
+func asyncTaskQueue(ctx context.Context, schema *arrow.Schema, recordStream 
chan<- RecordMessage, taskQueue <-chan taskState, h *cAsyncStreamHandler) {
        defer close(recordStream)
        for {
                select {
                case <-ctx.Done():
-                       C.goCallCancel(producer)
+                       h.mu.Lock()
+                       if h.producer != nil {
+                               C.goCallCancel(h.producer)
+                       }
+                       h.mu.Unlock()
                        return

Review Comment:
   Not from this PR, but related to the same spec section. On cancellation the 
goroutine calls `cancel` and returns at once, so tasks still in `taskQueue`, 
and any the producer sends afterwards for requests already made, are never 
passed to `extract_data`.
   
   The spec says a consumer that doesn't want a task's data "must still call 
`extract_data` so that the producer can perform any required cleanup", passing 
`NULL`. It also says the consumer must expect more `on_next_task` calls after 
`cancel`. So the producer never gets to clean those tasks up.
   
   Now that `asyncTaskExtract` handles a `NULL` `out`, the consumer side could 
drain the queue after `cancel` and call `extract_data` with `NULL` for each 
task until `release` closes the channel. That seems better as a follow-up than 
in this PR. I found this by reading the code and haven't run a conforming 
producer against it.



##########
arrow/cdata/cdata_exports.go:
##########
@@ -535,7 +543,11 @@ func asyncTaskQueue(ctx context.Context, schema 
*arrow.Schema, recordStream chan
                        }
 
                        // request another batch now that we've processed this 
one
-                       C.goCallRequest(producer, C.int64_t(1))
+                       h.mu.Lock()

Review Comment:
   A correction to the description. It says "the spec asks producers not to 
call `release` from inside `request` while not cancelled". I can't find that in 
the spec.
   
   - The `ArrowAsyncProducer.request` docs only forbid `request` from 
synchronously calling `on_next_task` or `on_error`.
   - `cancel` must not call any consumer handler other than `on_error`, so 
`release` is ruled out from inside `cancel`, but not from inside `request`.
   
   That matters because `h.mu` is held across this call. If a producer called 
`release` synchronously from inside `request`, the release callback would run 
on this goroutine and block on `h.mu` forever. A conforming producer is 
unlikely to do that, since it calls `release` after the end of the stream or an 
error. Could you either correct the spec claim, or list this case as a known 
deadlock next to the lock-ordering case the description already mentions?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to