davlee1972 commented on PR #50546: URL: https://github.com/apache/arrow/pull/50546#issuecomment-5037774622
I understand the complexity and for pure performance reasons I don't think you want to add Nulls in a non nullable array to begin with, but I'm forced to even out array lengths between parent records and child records. I added a function called fill_nulls_with_dummy() as a workaround to populate any non nullable array with zeros, empty string, 1970 dates, 0 duration, etc.. Maybe what we really need is a pyarrow.notnull() type, an option in pyarrow.compute.fill_null() to use a dummy fill value based on the values type and a pyarrow compute non_null_scalar() function that takes a type and returns back a dummy scalar. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
