aglinxinyuan commented on code in PR #5700: URL: https://github.com/apache/texera/pull/5700#discussion_r3565481973
########## amber/src/test/python/core/models/test_loop_operators.py: ########## @@ -0,0 +1,394 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Unit tests for the loop runtime: LoopStartOperator and LoopEndOperator. + +These exercise the abstract base classes in operator.py that the +generated `ProcessLoopStartOperator` / `ProcessLoopEndOperator` classes +extend. The tests use minimal stub subclasses that mirror what +`LoopStartOpDesc.generatePythonCode` / `LoopEndOpDesc.generatePythonCode` +emit so the behavior covered here is the same shape that ships at +runtime. + +Coverage: + - LoopStart's first-entry state merge into self.state. + - LoopEnd's process_table identity yield; condition is abstract. + - The guarded eval/exec helpers (eval_output / run_update / eval_condition) + keep the reserved `table` name out of the persistent loop state, so user + code cannot silently clobber loop machinery. The table crosses the loop + boundary as Arrow IPC bytes (see table_to_ipc_bytes in core.models.table). + - A multi-iteration loop driven to completion through the operators and the + State to_tuple/from_tuple round-trip (TestLoopRunsToCompletion). + +loop_counter and the LoopStart jump metadata (LoopStartId) are owned by the +worker runtime, not these operators -- they ride the StateFrame envelope as +their own columns (the loop-back write address is setup config, not state) -- +so their handling is covered in test_main_loop.py::TestMainLoop. +""" + +from typing import Iterator, Optional + +import pyarrow as pa +import pytest + +from core.models import State, Table, TableLike, Tuple +from core.models.operator import ( + _RESERVED_STATE_KEYS, + LoopEndOperator, + LoopStartOperator, +) +from core.models.table import table_from_ipc_bytes, table_to_ipc_bytes + + +# --------------------------------------------------------------------------- +# Stub subclasses that mirror the generated Python in +# LoopStart/LoopEnd OpDesc. Keeping them here (rather than reusing the +# real generator) lets the test pin behavior without spinning up a Scala +# runtime to produce code. +# --------------------------------------------------------------------------- + + +class _StubLoopStart(LoopStartOperator): Review Comment: Added `TestGeneratedCodeShape` (in `test_loop_operators.py`): it builds the `ProcessLoop{Start,End}Operator` source exactly as `generatePythonCode` emits it — user expressions delivered via `self.decode_python_template('<base64>')` — and `exec`s it with a quote and a newline in the expressions, then runs `open()`/`process_table` and `process_state()`/`condition()`. That exercises the base64/decode/exec-eval chain the stubs skip, catching codegen↔runtime drift without the integration job. (The templates mirror `generatePythonCode`; the Scala `*OpDescSpec` `code should include(...)` assertions pin the generated shape.) (5e1a29f) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
