GitHub user Xiao-zhen-Liu added a comment to the discussion: Proposal: 
operators that emit on more than one output port

Thanks for writing up the Phase 2 design before opening PRs, @sshiv012. Here 
are my thoughts on the five points:

1. **`PortOutput(value, port=1)` on the existing generator methods:** Agreed. 
Keeping one generator method per UDF base class with an explicit `PortOutput` 
wrapper is cleaner than adding separate `*_multi_port` methods (which would 
double the callback surface across the four UDF base classes and drop 
`@abstractmethod` checks on 1-port UDFs) or yielding raw `(row, port)` pairs.
2. **Unnamed yields on multi-port UDFs:** Keep bare `yield value` for 
1-output-port UDFs, and require `PortOutput(value, port=...)` on every output 
(including port `0`) whenever the operator has 2 or more output ports, raising 
a `ValueError` on a bare yield:
   ```python
   def process_tuple(self, tuple_: Tuple, port: int) -> 
Iterator[Optional[UDFOutput]]:
       if tuple_["latency_ms"] <= 1000:
           yield PortOutput(tuple_, port=0)
       else:
           yield PortOutput(tuple_, port=1)
   ```
   On the JVM side (`DataProcessor.outputOneTuple` -> 
`OutputManager.getSingleOutputPortIdentity`), an output with `None` port 
asserts `ports.size == 1` rather than broadcasting. Requiring `PortOutput` on 
multi-port Python UDFs matches that rule, reads symmetrically across ports, and 
turns a forgotten wrapper on a same-schema split into an immediate error 
instead of silently routing every row to port `0`.
3. **Zero-based `int` port indices:** Agreed. This matches the input `port: 
int` parameter and survives port renames. Please also reject `bool` values 
since `isinstance(True, int)` is `True` in Python.
4. **Runtime error on unknown ports:** Agreed. Please include the invalid port 
index and the operator's valid output port indices in the `ValueError` message.
5. **Per-port schemas (and visualization helper):** Using one shared schema 
across all output ports in the first Phase 2 PR is a good first step so routing 
lands on its own. For the second Phase 2 PR, rather than an interim `kind`-only 
setting that still forces all `table` ports to share operator-level columns 
(which cannot express cases like parsed rows on port `0`, reject rows with an 
`error` column on port `1`, and summary counts on port `2`), let us put the 
full output schema on each output port directly: `kind` (`table` with 
`SET_SNAPSHOT` vs `visualization` with `html-content: STRING` and 
`OutputMode.SINGLE_SNAPSHOT` in `PythonUDFOpDescV2.operatorInfo`), plus 
per-port `retainInputColumns` and `outputColumns` for `table` ports (falling 
back to the operator-level fields for saved workflows without per-port 
schemas). For `visualization` ports, a small helper such as 
`PortOutput.html(html, port=1)` (wrapping `{"html-content": html}`) would also 
avoid making users write the internal
  `"html-content"` column name by hand.

One implementation note for `OutputManager` (`output_manager.py`): `emit_ecm` 
and `emit_state` also call `tuple_to_frame` when flushing buffered batches, so 
please pass each link's `from_port_id` there as well once ports can have 
different schemas.

@Yicong-Huang could you also share your thoughts on this Python UDF multi-port 
API and per-port schema design before @sshiv012 opens the Phase 2 PRs?

GitHub link: 
https://github.com/apache/texera/discussions/8914#discussioncomment-19787161

----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]

Reply via email to