andygrove opened a new pull request, #70: URL: https://github.com/apache/datafusion-spark/pull/70
## Summary - Adds Spark's `array_insert(array, pos, item)`, ported from DataFusion Comet's `array_funcs/array_insert.rs` as a scalar UDF with type coercion. - 1-based index; negative counts from the end (`-1` appends). A position past the end pads with NULLs. A negative position before the start puts the item first, then NULL padding, then the original elements. - NULL array or NULL position gives NULL; a NULL item is inserted as a NULL element; index `0` fails with `INVALID_INDEX_OF_ZERO`. - `SparkArrayInsert::with_legacy_negative_index(true)` restores `spark.sql.legacy.negativeIndexInArrayInsert` behavior (`-1` inserts before the last element). Default is Spark 4's behavior. ## Differences from Comet's version - The size limit is checked before any NULL padding is allocated, so a huge index fails with `COLLECTION_SIZE_LIMIT_EXCEEDED` instead of allocating billions of NULLs. - Index arithmetic is 64-bit, so `i32::MIN` doesn't overflow. ## Behavior choices - Spark requires an INT index, but DataFusion types bare integer literals as BIGINT. Any integer type is accepted and cast to INT; non-integers are rejected. - The result's element field is always nullable, since padding can add NULLs even when neither input has any. - A NULL item takes the element type; numeric items widen to a common type; other mismatches are rejected. ## Testing - Unit tests plus `array/array_insert.slt`. Expected values come from the Spark docs and Comet's Spark-verified tests, not re-checked against a live Spark. - `cargo clippy --all-targets --all-features -- -D warnings`, `cargo test -p datafusion-spark` and `cargo test -p datafusion-spark-sqllogictest` pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
