aglinxinyuan opened a new issue, #5778:
URL: https://github.com/apache/texera/issues/5778
### Task Summary
Add a `TextGenCodegenSpec.scala` that pins the Python snippets generated for
the Hugging Face `text-generation` task — the OpenAI chat-completions payload
shape and the response-parsing path — plus the safety property that no
user-provided strings are inlined.
## Background
`TextGenCodegen` (`operator/huggingFace/codegen/TextGenCodegen.scala`) is a
`TaskCodegen` `object` that emits two Python snippets spliced into the shared
Hugging Face inference template: `payloadPython(ctx)` (builds the request
`payload`) and `parsePython(ctx)` (extracts the model output). It is the
default/fallback codegen in `HuggingFaceInferenceOpDesc.registeredCodegens`,
and has no dedicated unit-spec.
## Behavior to pin
| Surface | Contract |
| --- | --- |
| `TextGenCodegen.task` | equals `"text-generation"` |
| `payloadPython(ctx)` | opens with `if task == "text-generation":`; builds
a `payload` dict with a `messages` list (a `system` + a `user` message),
`max_tokens`, and `temperature`; includes the `else: payload = {"inputs":
prompt_value}` fallback |
| `parsePython(ctx)` | returns the branch reading
`body["choices"][0]["message"]["content"]` |
| no raw-string leakage | both snippets reference the per-instance
attributes (`self.MODEL_ID`, `self.SYSTEM_PROMPT`, `self.MAX_NEW_TOKENS`,
`self.TEMPERATURE`) and loop locals (`prompt_value`, `task`) — they must NOT
contain the literal values from the passed `CodegenContext` |
| context-independent | the text-generation snippets ignore `ctx`: two
different `CodegenContext`s produce identical output |
`CodegenContext`'s string fields are `EncodableString` (a `String`
type-alias), so plain string literals work in a test. See
`HuggingFaceInferenceOpDesc` (~line 164) for a `CodegenContext(...)`
construction example.
## Scope
- New spec: `TextGenCodegenSpec.scala` under
`common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/huggingFace/codegen/`.
- Assert on snippet content with substring/contains checks (the snippets are
deterministic Python text); no Python execution or engine needed.
- No production-code changes.
### Task Type
- [ ] Refactor / Cleanup
- [ ] DevOps / Deployment / CI
- [x] Testing / QA
- [ ] Documentation
- [ ] Performance
- [ ] Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
