svilupp opened a new issue, #403:
URL: https://github.com/apache/arrow-julia/issues/403
This is an odd one and likely to be a PICNIC...
Problem: Missigness in a string column is lost after saving/loading arrow
file
When it happens: When a column in my dataset has type
`Union{Missing,String}`, I partition it, and the missing item appears only in
the later partitions. It's easily reproducible (see below).
Debugging:
- It happens only to DataFrames (not Tables.rowtable when created from a
namedtuple)
- Only when partitioned as `Iterators.partition(Tables.rows(df), 2)`. If
partitioned as `Iterators.partition(df,2)` [available from version
>1.5.0](https://github.com/JuliaData/DataFrames.jl/pull/3212), it is fine
- If missing type appears in the first partition, it's fine
- Validity bitmap is written correctly
- But field is marked as not-nullable (!)
> ┌ Debug: building field: name = x1, nullable = false, T = String, type =
Arrow.Flatbuf.Utf8
>└ @ Arrow ~/Documents/GitHub/arrow-julia/src/write.jl:486
> --- in correct cases, this appears
> ┌ Debug: building field: name = x1, nullable = true, T = Union{Missing,
String}, type = Arrow.Flatbuf.Utf8
> └ @ Arrow ~/Documents/GitHub/arrow-julia/src/write.jl:486
MWE
```
using Arrow, Tables, Random, DataFramesMeta
using Logging
debuglogger = ConsoleLogger(stderr, Logging.Debug)
# Create dataset
fn = "test_types.arrow"
df = Tables.rowtable((; x1 =["a","b",missing,"c"], x2 = 1:4)) |> DataFrame
# Works okay
Arrow.write(fn, df; compress = nothing);
t=Arrow.Table(fn)
t[:x1]
# Arrow.List{Union{Missing, String}, Int32, Vector{UInt8}}
# Works okay
Arrow.write(fn, Iterators.partition(df,2); compress = nothing);
t=Arrow.Table(fn)
t[:x1]
# SentinelArrays.ChainedVector{Union{Missing, String},
Arrow.List{Union{Missing, String}, Int32, Vector{UInt8}}}:
# broken -- missingness is lost
Arrow.write(fn, Iterators.partition(Tables.rows(df), 2); compress = nothing);
t=Arrow.Table(fn)
t[:x1]
# SentinelArrays.ChainedVector{String, Arrow.List{String, Int32,
Vector{UInt8}}}
# Works okay with Tables
t = Tables.rowtable((; x1 =["a","b",missing,"c"], x2 = 1:4))
Arrow.write(fn, Iterators.partition(Tables.rows(t), 2); compress = nothing);
t=Arrow.Table(fn)
t[:x1]
# SentinelArrays.ChainedVector{Union{Missing, String},
Arrow.List{Union{Missing, String}, Int32, Vector{UInt8}}}
```
Versioninfo:
> Julia Version 1.8.5
> Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
> Platform Info:
> OS: macOS (arm64-apple-darwin21.5.0)
> CPU: 8 × Apple M1 Pro
> WORD_SIZE: 64
> LIBM: libopenlibm
> LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
> Threads: 6 on 6 virtual cores
> Arrow: 2.4.3 on main branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]