eugenegujing opened a new issue, #7746:
URL: https://github.com/apache/texera/issues/7746
### What happened?
The worker service and the transport envelope disagree about
EvaluatePythonExpression's reply type, and the loser is silently discarded.
`workerservice.proto:51` declares the worker's reply as `EvaluatedValue`:
```proto
rpc EvaluatePythonExpression(EvaluatePythonExpressionRequest) returns
(EvaluatedValue);
```
but every worker reply must travel inside `ControlReturn`'s sealed oneof
(`controlreturns.proto`), and that oneof registers only the coordinator-service
wrapper — `EvaluatePythonExpressionResponse` at `controlreturns.proto:39`,
which is the reply type of the *coordinator's* RPC
(`coordinatorservice.proto:37`) and is `repeated EvaluatedValue values`
(`controlreturns.proto:113-114`). `EvaluatedValue` itself is defined at
`controlreturns.proto:108` but never joins the oneof.
On the Python engine the consequence is a silent drop, not an error:
`set_one_of` ignores a type that is not a oneof member, so the worker's reply
is packed into an **empty** `ControlReturn`.
```python
packed = set_one_of(ControlReturn,
EvaluatedValue(value=TypedValue(expression="1+1", value_str="2")))
bytes(packed) # b'' -- nothing on the wire
get_one_of(packed) # None -- nothing to unpack
```
A control case shows registered types survive the same path:
`set_one_of(ControlReturn, WorkerStateResponse())` → `b'\x92\x03\x00'` and
`get_one_of` returns the message.
The path is user-facing. The workflow console's expression evaluator sends
`EvaluatePythonExpressionRequest` (`ExecutionConsoleService.scala:295`); the
coordinator handler fans out to every worker and `Future.collect`s their
`EvaluatedValue` replies to build the response
(`EvaluatePythonExpressionHandler.scala:37-55`); the Python worker's handler
(`core/architecture/handlers/control/evaluate_expression_handler.py`) does
produce an `EvaluatedValue` — which then cannot cross the wire back.
The swallowing *mechanism* is already pinned by
`test_async_rpc_server.py:262-280` with a synthetic wrong type; this issue is
about the one real RPC whose declared reply type hits it. The proto asymmetry
is language-neutral (both engines generate from the same files); only the
Python packing behavior has been verified.
**Expected:** the evaluated value reaches the coordinator. The fix is a
.proto change: register `EvaluatedValue` in `ControlReturn`'s sealed oneof
(verified locally that this closes the hole — packed bytes become non-empty and
`get_one_of` returns the value), or change `workerservice.proto` to reply with
the wrapper type. Either way both engines regenerate.
### How to reproduce?
Run against `amber/src/main/python`:
```python
from core.util import set_one_of, get_one_of
from proto.org.apache.texera.amber.engine.architecture.rpc import (
ControlReturn,
EvaluatedValue,
TypedValue,
)
packed = set_one_of(
ControlReturn, EvaluatedValue(value=TypedValue(expression="1+1",
value_str="2"))
)
print(bytes(packed)) # b''
print(get_one_of(packed)) # None
```
The verified output is in the log section below.
### Version/Branch
1.3.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
e80add40c
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
bytes(set_one_of(ControlReturn, EvaluatedValue(...))) == b''
get_one_of(set_one_of(ControlReturn, EvaluatedValue(...))) is None
control case: bytes(set_one_of(ControlReturn, WorkerStateResponse())) ==
b'\x92\x03\x00'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
