aglinxinyuan commented on PR #5700: URL: https://github.com/apache/texera/pull/5700#issuecomment-4910871794
### Serialization design for loop state Sharing this per @chenlica's request, with context from an offline discussion with @Yicong-Huang: Arrow's columnar format isn't always faster than pickle on data transfer — nested types have been ~3x slower for years (it resurfaced in the Spark 4.2.0 "arrow-by-default for Python UDFs" work). That's directly relevant to how a loop serializes the state it passes from Loop Start to Loop End, so here's the design choice and how it evolved. **What the state is.** Each iteration, Loop Start hands Loop End a "state" — the user's loop variables (`i`, accumulators, …) plus the input `table`. It isn't an in-memory hand-off: the state is written to the cross-region iceberg state channel (#4490) and read back by a *different* worker (the next iteration's Loop Start, via the back-edge). **Why pickle first.** The state is a mixed/nested payload and can hold arbitrary Python objects — including a single large binary such as an ML model. Arrow buys nothing there: it's slower for nested types, and for a low-volume, same-schema state (or one big blob) the columnar layout doesn't help — as @Yicong-Huang noted. `pickle` round-trips arbitrary objects with zero schema plumbing, so it was the pragmatic first cut. **Why we moved off pickle.** Because the state is persisted and then deserialized on another worker, `pickle.loads` on those stored bytes is a remote-code-execution surface — a tampered or malicious state document would execute arbitrary code on load. That's not acceptable for data that round-trips through shared storage. So the shipped design uses no pickle: - the `table` is serialized as an **Apache Arrow IPC stream** (`table_to_ipc_bytes` / `table_from_ipc_bytes` in `core/models/table.py`) — a length-prefixed, schema-typed, data-only container that cannot execute code on load; - the loop variables ride the State `content` column as **JSON**, with any raw `bytes` (e.g. the model blob) base64-encoded via a type marker. **On the performance concern.** The Arrow nested-type penalty doesn't bite here: Texera tables are flat (no nested column types), so the table is a straight flat-columnar encode/decode. The genuinely arbitrary/large bits (loop vars, a model blob) go through JSON + base64, not Arrow, so they're a plain byte copy — and the state channel is low-volume, so the cost is negligible next to the safety guarantee. Net: pickle would have been simpler and perfectly fine on performance, but it's an RCE liability across persisted/shared storage — so loop state is Arrow IPC (table) + JSON/base64 (everything else), and deliberately no pickle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
