PG1204 opened a new issue, #7906:
URL: https://github.com/apache/texera/issues/7906
### What happened?
When the Hugging Face Inference operator falls back from `hf-inference` to a
third-party chat-completions provider, the image tasks degrade in two ways.
This is the same bug class as #7195, in `ImageTaskCodegen` instead of
`QaRankingCodegen`.
**1. Candidate labels are dropped from the request.** For
`zero-shot-image-classification`, `ImageTaskCodegen.payloadPython` (L92-100)
builds the native HF pipeline payload:
payload = {"inputs": <base64 image>, "parameters": {"candidate_labels":
labels}}
Only `hf-inference` understands that shape. On a third-party provider,
`_call_provider` rebuilds the request as an image part plus a text part
(`HuggingFaceCodegenBase.scala` L369, L511, L532), where the text is just
`prompt_value`. `parameters` is never carried over, so the provider receives an
image with no candidate labels: the input that defines the task. The result is
a generic caption instead of a classification, with no error reported.
`visual-question-answering` and `document-question-answering` are unaffected
here, since their question is `prompt_value`, which the chat branch does send.
**2. Chat responses are written to the result column as raw JSON.** Chat
providers reply with `{"choices": [{"message": {"content": ...}}]}`, and three
image tasks cannot read that shape in `ImageTaskCodegen.parsePython`:
- `zero-shot-image-classification` (L147) is grouped with three other tasks
and always returns `json.dumps(body)`.
- `visual-question-answering` and `document-question-answering` (L114)
return `body.get("answer", json.dumps(body))`, and a chat response has no
`answer` key.
So even when the provider answers correctly, the user sees the raw JSON
envelope instead of the answer. `image-to-text` and `image-text-to-text` in the
same file already handle `choices` correctly, so the fix is to apply that
existing idiom to the remaining three tasks.
Both halves need fixing together: correcting only the request still displays
raw JSON, and correcting only the response cleanly displays an answer to a
question the model was never asked.
The same raw-JSON output appears for `visual-question-answering` and
`document-question-answering` whenever they fall back to a chat provider.
### How to reproduce?
1. Add a Hugging Face Inference operator with task
`zero-shot-image-classification`, an image column, and two or more Candidate
Labels.
2. Choose a model whose cheapest provider is not `hf-inference`, or make
`hf-inference` fail, so the request goes through `_call_provider`'s chat branch.
3. Run the workflow. The outbound request carries the image and the prompt
text but no candidate labels, and the result column contains a raw chat JSON
envelope rather than a label.
### Version/Branch
1.3.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
_No response_
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
