[
https://issues.apache.org/jira/browse/SPARK-59349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh reassigned SPARK-59349:
-----------------------------------
Assignee: L. C. Hsieh
> Override the gRPC authority for UDF worker Unix socket channels
> ---------------------------------------------------------------
>
> Key: SPARK-59349
> URL: https://issues.apache.org/jira/browse/SPARK-59349
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> {{GrpcWorkerChannel}} builds its gRPC channel without calling
> {{overrideAuthority}}, so gRPC derives the HTTP/2 {{:authority}}
> pseudo-header from the target address. For a Unix domain socket that address
> is the socket path, which is not a valid HTTP authority (and is
> percent-encoded where the path needs escaping, e.g.
> {{var%2Ffolders%2F...%2Frw.sock}}).
> A conforming HTTP/2 server validates the pseudo-headers while decoding the
> HEADERS frame and resets the stream:
> {noformat}
> h2::server: malformed headers: malformed authority (...): invalid authority
> h2::proto::streams::send: send_reset(..., reason=PROTOCOL_ERROR,
> initiator=Library, ...)
> {noformat}
> The client sees only:
> {noformat}
> io.grpc.StatusRuntimeException: INTERNAL: RST_STREAM closed stream. HTTP/2
> error code: PROTOCOL_ERROR
> {noformat}
> The rejection happens before any worker application code runs -- not the
> service method, not an interceptor -- so the worker logs nothing and Init is
> never observed. The failure points at the wrong side of the connection, and
> the worker has no way to report why.
> This is not reachable through any in-tree code path today. grpc-java's own
> server tolerates the malformed authority, and both Spark's client and the
> only in-tree worker ({{EchoGrpcWorkerMain}}) are grpc-java, so the pair works
> and the header is never validated. That is why it went unnoticed in
> SPARK-56922.
> It does affect any worker written against a conforming HTTP/2 stack, which is
> most non-JVM stacks. worker_spec.proto invites workers in any language and
> the SPARK-55278 SPIP names onboarding a Go/Rust/Swift worker as its final
> exam; such a worker hits this on its very first RPC. Found while building a
> Rust worker, and confirmed three ways: the same client succeeds against the
> same worker binary when only the authority changes; driving the worker
> through Spark's own DirectGrpcDispatcher reproduces the identical
> PROTOCOL_ERROR (so it is not a grpc-python quirk); and the server-side h2
> trace names the rejected header.
> Fix: call {{overrideAuthority}} on the channel builder with a placeholder
> authority. A placeholder is correct rather than a workaround -- for a Unix
> socket peer the authority carries no information, because the socket path
> already identifies the peer completely. There is no name to resolve and no
> virtual host to select.
> Note this cannot be fixed on the worker side: a tonic/tower layer rewriting
> the request URI runs after header decode, by which point the stream is
> already reset.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
