mkzung commented on issue #37476: URL: https://github.com/apache/arrow/issues/37476#issuecomment-4945708067
Ran into this from the narwhals side: its `Categorical` maps to `dictionary(uint32, string)`, so building one from Python values fails there. I opened #50474 before I spotted this issue, and I've closed that as a duplicate of this one. One thing that might be worth adding here: it isn't only the signedness that gets dropped. The index type is really treated as a *starting bit width*, so even a signed type stops being honoured once the cardinality outgrows it: ```py >>> import pyarrow as pa, string, itertools >>> vals = ["".join(p) for p in itertools.islice(itertools.product(string.ascii_lowercase, repeat=2), 200)] >>> pa.array(["a", "b", "a"], type=pa.dictionary(pa.int8(), pa.string())).type.index_type DataType(int8) >>> pa.array(vals, type=pa.dictionary(pa.int8(), pa.string())).type.index_type DataType(int16) ``` That falls out of the same place @jorisvandenbossche pointed at: `MakeDictionaryBuilder` passes `exact_index_type=false`, and `DictionaryBuilderCase::CreateFor()` reduces the requested type to `index_type->byte_width()` and hands that to `AdaptiveIntBuilder` as a starting width. So the signedness is lost and the width is only a hint. I'd like to try the fix along the lines Joris suggested (an adaptive *unsigned* index builder for unsigned index types, keeping the width adaptive). Two things I hit while reading around it: - `AdaptiveUIntBuilder`'s constructors don't take the `alignment` argument that `AdaptiveIntBuilder`'s do, so `DictionaryBuilderBase<AdaptiveUIntBuilder, T>` doesn't compile as it stands. - `DictionaryConverter` in `util/converter.h` hardcodes `DictionaryBuilder<ValueType>` and `checked_cast`s to it, and that trait is shared with the R binding, so it looks like it needs an index-builder template parameter rather than a local change. Happy to pick this up if nobody else is on it. Versions: pyarrow 23.0.1, Python 3.10, macOS arm64. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
