walterddr opened a new pull request, #10019:
URL: https://github.com/apache/pinot/pull/10019
This adds GRPC mailbox offer timeout.
Details
===
when GRPC `StreamObserver.onNext()` is called (triggered by mailbox sender
to receiver, via GRPC channels), several things happens:
- "The StreamObservers returned by gRPC from the stub are async.
StreamObservers themselves don't have to be async, so one could implement one
that blocks"
- "Since the callbacks for the gRPC are considered non-thread-safe,
blocking StreamObservers will delay other callbacks until `onNext()` returns.
- "That includes `setOnReadyHandler` and `setOnCancelHandler` in
`ClientCallStreamObserver` and `ServerCallStreamObserver`."
- "In streaming RPCs, gRPC automatically requests another message after
StreamObserver returns from `onNext()`, so if blocked, gRPC will avoid
receiving more messages. gRPC will still allow some messages to be buffered,
- "however, Blocking only affects the current streaming RPC. It has no
impact on new RPCs."
This means
1. waiting for the process to finish after receiving `onNext()` will create
a "poor-man" back-pressure b/c gRPC
- it will not process another message until `onNext()` returns from the
receiving end.
- it will not drop messages.
- implicitly, i guess, this means the the sender side invocation to the
`streamObserver.onNext()` is also blocked (via GRPC internal mechanism)
2. ***[TODO]*** when receiving arrayBlockingQueue buffer is full and offer
times out, it will return false and should log and exception
- if we use the same timeout for the entire mailbox. it should thrown an
exception explicitly at this point from outer mailbox abstraction
- we should allow changing of timeout via QueryOptions.
- if we decided to use a much smaller timeout (e.g. smaller timeout for
a single msg, vs. larger timeout for the entire mailbox); we should log the
exception and attach the exception to the last end-of-stream message indicating
a warning that results might be incomplete and suggest increase timeout
Reference
===
- https://grpc.github.io/grpc-java/javadoc/io/grpc/stub/StreamObserver.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
