davlee1972 opened a new issue, #50515:
URL: https://github.com/apache/arrow/issues/50515
### Describe the bug, including details regarding any error messages,
version, and platform.
Is this a bug and if it isn't is there a preferred way of handling this?
I've got some rather complex data structures I'm trying to save into parquet
using a common schema and using cast() on a struct array would be the preferred
way to align data.
Shouldn't NULL in an inner ray be acceptable when it's validity mask is set
to False? I know I can call pyarrow.compute.fill_null() and replace the NULL
with a ZERO, but this can get complicated when I got thousands of fields with a
mix of all kinds of datatypes: int, string, decimal, date, timestamp, boolean,
etc.. and there isn't a clear way to stuff dummy scalars: pc.scalar(0),
pc.scalar(""), pc.scalar(True), etc.. for every possible datatype to replace
NULL.
Here are working and non-working example using cast() on a the same struct
array type.
Define struct types and setup data using a pylist vs a list of arrays.
<img width="1215" height="640" alt="Image"
src="https://github.com/user-attachments/assets/06a7b3d7-1fd3-4841-b320-308174aadf59";
/>
If we examine the two struct, to_pylist() returns the same values, but the
inner struct for struct a contains a "0" while struct b has a "null".
<img width="1074" height="1006" alt="Image"
src="https://github.com/user-attachments/assets/995a794a-67c7-4a28-b844-cadba26d45f0";
/>
Casting both structs to the same outer struct type fails for struct b
because it has a NULL value in it even though the field it belongs to in the
outer struct is nullable.
<img width="1183" height="948" alt="Image"
src="https://github.com/user-attachments/assets/c1810ca1-3725-40e7-9c2a-34778c42d2d6";
/>
### Component(s)
C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
