I came up with a workaround, but it seems like a lot of extra work to get a
child array to fit into its parent.
Populate valid child values into xpath_elem.data_pyarrow.
Track parent positions in xpath_elem.data_offsets. This includes using None for
parent positions with no children.
Run data validation rules on:
xpath_elem.data_pyarrow
using:
xsd restrictions converted into pyarrow.compute expressions.
pydantic rules
Rebuild child array after validation using pyarrow.compute.take()
xpath_elem.data_pyarrow = pc.take(xpath_elem.data_pyarrow,
pa.array(xpath_elem.data_offsets, type=pa.int64()))
Fill nulls in array created by pyarrow.compute.take() with dummy values:
if not xpath_elem.nullable and xpath_elem.data_pyarrow.null_count:
xpath_elem.data_pyarrow = fill_nulls_with_dummy(
xpath_elem.data_pyarrow
)
Dummy fill function:
def fill_nulls_with_dummy(arr: pa.Array) -> pa.Array:
"""Fills null values in a pyarrow array with dummy scalar values based on
its type."""
data_type = arr.type
# Map pyarrow types to dummy Python values
if pa.types.is_boolean(data_type):
dummy_val = False
elif (
pa.types.is_integer(data_type)
or pa.types.is_floating(data_type)
or pa.types.is_decimal(data_type)
or pa.types.is_duration(data_type)
):
dummy_val = 0
elif pa.types.is_string(data_type) or pa.types.is_binary(data_type):
dummy_val = ""
elif pa.types.is_timestamp(data_type):
dummy_val = datetime(1970, 1, 1)
elif pa.types.is_date(data_type):
dummy_val = date(1970, 1, 1)
elif pa.types.is_list(data_type):
dummy_val = []
elif pa.types.is_struct(data_type):
dummy_val = {}
else:
raise ValueError(str(data_type) + "is not supported yet.")
# Create the typed scalar and apply the fill null compute function
scalar = pa.scalar(dummy_val, type=data_type)
return pc.fill_null(arr, scalar)
This message may contain information that is confidential or privileged. If you
are not the intended recipient, please advise the sender immediately and delete
this message. See
https://www.blackrock.com/corporate/compliance/email-disclaimers for further
information. Please refer to
https://www.blackrock.com/corporate/compliance/privacy-policy for more
information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see
https://www.blackrock.com/corporate/about-us/contacts-locations.
© 2026 BlackRock, Inc. All rights reserved.
