jonasdedden commented on PR #51611:
URL: https://github.com/apache/arrow/pull/51611#issuecomment-6395800659
Since the type issues kept coming one at a time, I fuzzed `list_contains`
for results that change when only the representation of the same data changes.
This covered about 30 value types * 30 query types:
- **Values:** dictionary encoding, plain/large/view layouts, child offsets,
garbage values under null lists or between views.
- **Lists:** list, large list, list view and fixed-size list layouts, slices
and chunks.
- **Queries:** scalar vs array queries, dictionary encoding and chunks.
- **Code paths:** scalar lists vs repeated lists.
- **Checks:** same-type results are compared against searching the decoded
lists, and type errors must not depend on the data.
Against an older build it finds all the issues raised in review so far.
Against c47a14b it found these:
- **Binary views vs other binary layouts:** `equal` can't compare `string`
with `string_view`, so `list<string>` with a `string_view` query raised, while
a `string` query worked. Views are now compared as their large types, like
float16 as float32.
- **Temporal dictionaries without casts:** the cast check looked at the
dictionary type, which accepts every cast. So a dictionary of durations against
timestamps returned a result for null queries (`[False]`) and raised
`ArrowInvalid` otherwise, while plain durations raise `NotImplemented`. It now
checks the decoded type.
- **All-null dictionary queries:** queries of a `null`-typed dictionary, or
a dictionary with only null entries, weren't recognized as null queries. They
raised against values that `equal` can't compare, like structs, while a plain
null query works. Null queries are now detected after decoding dictionaries.
- **Null-typed dictionaries segfaulted:** `dict_util` read the validity
bitmap of a `null`-typed dictionary, which has none. This crashed
`list_contains` with null queries on a scalar list of such values, and on
`main` it already crashes `pc.is_null` and `pc.is_valid`. Fixed in
`dict_util.cc` in its own commit (abc2fa2); the rest is in 260a708.
<details>
<summary>Reproduction</summary>
```python
import datetime as dt
import pyarrow as pa
import pyarrow.compute as pc
def show(label, call):
try:
print(f"{label}: {call().to_pylist()}", flush=True)
except pa.ArrowException as e:
print(f"{label}: {type(e).__name__}: {str(e)[:80]}", flush=True)
# 1. Binary views vs the other binary layouts
strings = pa.array([["x"], ["y"]])
show("1a. list<string> vs string query, expect [True, False]",
lambda: pc.list_contains(strings, pa.scalar("x")))
show("1b. list<string> vs string_view query, expect [True, False]",
lambda: pc.list_contains(strings, pa.scalar("x", pa.string_view())))
# 2. Temporal dictionaries vs durations, which timestamps don't cast to
timestamps = pa.array([[dt.datetime(2020, 1, 1)]],
pa.list_(pa.timestamp("s")))
durations = pa.array([None], pa.duration("s"))
show("2a. duration null query, expect NotImplemented",
lambda: pc.list_contains(timestamps, durations))
show("2b. dictionary duration null query, expect NotImplemented",
lambda: pc.list_contains(timestamps, durations.dictionary_encode()))
show("2c. dictionary duration query, expect NotImplemented",
lambda: pc.list_contains(timestamps, pa.array([dt.timedelta(1)],
pa.duration("s"))
.dictionary_encode()))
# 3. All-null dictionary queries vs values "equal" can't compare
structs = pa.array([[{"a": 1}, None]])
show("3a. null query, expect [True]", lambda: pc.list_contains(structs,
None))
show("3b. dict<null> null query, expect [True]",
lambda: pc.list_contains(structs, pa.DictionaryArray.from_arrays(
pa.array([0], pa.int8()), pa.nulls(1))))
# 4. Null-typed dictionaries (last, as this segfaults)
null_dict = pa.DictionaryArray.from_arrays(pa.array([0, 0], pa.int16()),
pa.nulls(1))
show("4a. scalar list of dict<null>, null queries, expect [True, True]",
lambda: pc.list_contains(pa.ListArray.from_arrays([0, 2], null_dict)[0],
pa.array([None, None], pa.int64())))
show("4b. is_null(dict<null>), expect [True, True]", lambda:
pc.is_null(null_dict))
```
On c47a14b:
```
1a. list<string> vs string query, expect [True, False]: [True, False]
1b. list<string> vs string_view query, expect [True, False]:
ArrowNotImplementedError: Function 'equal' has no kernel matching input types
(string, string_view)
2a. duration null query, expect NotImplemented: ArrowNotImplementedError:
Unsupported cast from duration[s] to timestamp[s]
2b. dictionary duration null query, expect NotImplemented: [False]
2c. dictionary duration query, expect NotImplemented: ArrowInvalid: Cast
type timestamp[s] incompatible with dictionary type duration[s]
3a. null query, expect [True]: [True]
3b. dict<null> null query, expect [True]: ArrowNotImplementedError: Function
'equal' has no kernel matching input types (struct<a: int64>, struct<a:
Segmentation fault
```
With abc2fa2 and 260a708, every line prints its expected result.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
