kz930 opened a new issue, #7549:
URL: https://github.com/apache/texera/issues/7549
### What happened?
Sentiment Analysis, Spam SMS Detection and Text Summarization all fail when
the text column they are configured on is empty in any row. Each passes the
field straight to a tokenizer or pipeline with no null check:
- `HuggingFaceSentimentAnalysisOpDesc.scala:84` —
`self.tokenizer(tuple_[attribute], return_tensors='pt')`
- `HuggingFaceSpamSMSDetectionOpDesc.scala:69` —
`self.pipeline(tuple_[attribute])[0]`
- `HuggingFaceTextSummarizationOpDesc.scala:66-68` — `text =
tuple_[attribute]`, then `self.tokenizer([text], ...)`
An empty value is ordinary input. A blank CSV cell arrives as null, since
univocity returns null for an empty field and `AttributeTypeUtils.parseField`
passes it through by design, its first line being `if (field == null) return
null`. It reaches the Python operator as `None`.
The messages differ because two go through `pipeline()` and one calls the
tokenizer directly, but the cause is the same in all three. Transformers
reports `ValueError: You need to specify either text or text_target` for the
first two, and `ValueError: text input must be of type str (single example),
list[str] (batch or single pretokenized example) or list[list[str]]` for the
third.
Expected: a row whose text column is empty either passes through with an
empty result, or is skipped, or fails with a message naming the column.
### How to reproduce?
Upload a CSV with a blank cell:
```
id,text
1,I absolutely love this
2,
3,this is the worst
```
Build `CSV File Scan` to `Sentiment Analysis`, point it at `text`, and run.
Repeat with `Spam SMS Detection` and with `Text Summarization` on the same
column.
### Version/Branch
1.3.0-incubating-SNAPSHOT (main)
### Relevant log output
```shell
File "<string>", line 18, in process_tuple
File ".../transformers/tokenization_utils_base.py", line 2542, in __call__
ValueError: You need to specify either `text` or `text_target`.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
