alamb commented on issue #7636: URL: https://github.com/apache/arrow-datafusion/issues/7636#issuecomment-1737225422
I think `row_number()` is marked as non-nullable because the implementation always produces a value (aka no nulls) and thus changing it to be nullable doesn't seem correct In the example above, running ```sql select * from customers ``` The output is this (there are no nulls): ``` +----+---------+---------+ | id | name | row_num | +----+---------+---------+ | 1 | Alice | 1 | | 2 | Bob | 2 | | 3 | Charlie | 3 | | 4 | David | 4 | | 5 | Eve | 5 | +----+---------+---------+ ``` I caught the error in the debugger and the error seems to be thrown in `concat_batches` called from here: https://github.com/apache/arrow-datafusion/blob/32dfbb0bb5c51b55ad84eed4fc140840f0a76612/datafusion/physical-plan/src/joins/cross_join.rs#L138 it turns out that this exact check was removed in arrow `47.0.0` https://github.com/apache/arrow-rs/pull/4815 (merged into DataFusion 2 days ago in https://github.com/apache/arrow-datafusion/pull/7587) and so when I run this query against the latest version of DataFusion (not yet released to crates.io) I get: ``` +----+---------+---------+----+---------+---------+ | id | name | row_num | id | name | row_num | +----+---------+---------+----+---------+---------+ | 1 | Alice | 1 | 1 | Alice | 1 | | 1 | Alice | 1 | 2 | Bob | 2 | | 1 | Alice | 1 | 3 | Charlie | 3 | | 1 | Alice | 1 | 4 | David | 4 | | 2 | Bob | 2 | 1 | Alice | 1 | | 2 | Bob | 2 | 2 | Bob | 2 | | 2 | Bob | 2 | 3 | Charlie | 3 | | 2 | Bob | 2 | 4 | David | 4 | | 3 | Charlie | 3 | 1 | Alice | 1 | | 3 | Charlie | 3 | 2 | Bob | 2 | | 3 | Charlie | 3 | 3 | Charlie | 3 | | 3 | Charlie | 3 | 4 | David | 4 | | 4 | David | 4 | 1 | Alice | 1 | | 4 | David | 4 | 2 | Bob | 2 | | 4 | David | 4 | 3 | Charlie | 3 | | 4 | David | 4 | 4 | David | 4 | | 5 | Eve | 5 | 1 | Alice | 1 | | 5 | Eve | 5 | 2 | Bob | 2 | | 5 | Eve | 5 | 3 | Charlie | 3 | | 5 | Eve | 5 | 4 | David | 4 | | 1 | Alice | 1 | 5 | Eve | 5 | | 2 | Bob | 2 | 5 | Eve | 5 | | 3 | Charlie | 3 | 5 | Eve | 5 | | 4 | David | 4 | 5 | Eve | 5 | | 5 | Eve | 5 | 5 | Eve | 5 | +----+---------+---------+----+---------+---------+ ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
