timsaucer commented on issue #715:
URL:
https://github.com/apache/datafusion-python/issues/715#issuecomment-2132233093
Closing in favor of https://github.com/apache/arrow/issues/41833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on
timsaucer closed issue #715: Selecting struct field within field produces
unexpected results
URL: https://github.com/apache/datafusion-python/issues/715
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
timsaucer commented on issue #715:
URL:
https://github.com/apache/datafusion-python/issues/715#issuecomment-2132211293
In my gist above, I went back an inserted values into the subfields
`inner_1` and `inner_2` even though `outer` was null and I *am* able to
reproduce the problem above, so
timsaucer commented on issue #715:
URL:
https://github.com/apache/datafusion-python/issues/715#issuecomment-2132210685
I think I know what's going on.
Even if `outer` is null, we still have data within `inner_1` and `inner_2`.
When pyarrow creates the record batch, it sets these to t
timsaucer commented on issue #715:
URL:
https://github.com/apache/datafusion-python/issues/715#issuecomment-2132207121
Further testing on the rust side makes me think it is something about how
the batch record is created in pyarrow. I created the same dataframe using
StructBuilder in the b
timsaucer commented on issue #715:
URL:
https://github.com/apache/datafusion-python/issues/715#issuecomment-2131229316
My statement above about testing on rust side is likely incorrect. I ran the
same test above but loading the dataframe from a parquet file instead of
creating in memory an
timsaucer opened a new issue, #715:
URL: https://github.com/apache/datafusion-python/issues/715
**Describe the bug**
When you have a column that is a struct of struct and you attempt to index
into the lowest level, if there is a null at the first level of the struct you
get an unexpected
