weimingdiit opened a new issue, #2419:
URL: https://github.com/apache/auron/issues/2419
**Is your feature request related to a problem? Please describe.**
Native joins use `RowNullChecker` to detect whether precomputed Arrow row
keys contain nulls. The result is later used by `StreamCursor.is_null_key()` to
decide join-key null handling.
The current implementation manually parses Arrow row bytes using per-type
length estimates:
- `Struct` uses a fixed placeholder length of `32`.
- `Dictionary` estimates length from the value type instead of the encoded
row/key representation.
- `List` / `LargeList` are treated as variable-length byte blocks.
- Unsupported Arrow types panic during `RowNullChecker` construction.
This looks fragile for complex join keys. If the computed field length is
wrong, the next key field may be read from the wrong offset, which can cause
either false null detection or missed null detection. That can lead to
incorrect native join results, especially for joins whose keys include nested
or dictionary-encoded types.
Relevant code:
- `native-engine/datafusion-ext-plans/src/joins/stream_cursor.rs`
- `StreamCursor::try_new()` builds `RowNullChecker`
- `StreamCursor::next()` computes `key_has_nulls`
- `StreamCursor::is_null_key()` consumes the result
- `native-engine/datafusion-ext-plans/src/common/row_null_checker.rs`
- `RowNullChecker::create_field_config_from_data_type`
- `RowNullChecker::calculate_field_length`
**Describe the solution you'd like**
`RowNullChecker` should not rely on approximate or hardcoded byte lengths
for Arrow row encoding.
Possible fixes:
- Derive null-key information from the original key arrays before converting
them into Arrow `Rows`.
- Or update `RowNullChecker` to parse Arrow row encoding accurately for all
supported key types.
- Replace panic paths with `Result` errors so unsupported key types can fall
back cleanly instead of crashing.
- Add coverage for join keys containing nested/complex types, including:
- `Struct`
- `List` / `LargeList`
- `Dictionary`
- `Decimal`
- mixed null/non-null key fields
- different `SortOptions`, including `nulls_first` and descending order
**Additional context**
The suspicious parts are:
```rust
DataType::Struct(_) => {
// For struct types, we need to calculate the total encoded length
// This is a simplified implementation
let estimated_length = 32; // Placeholder for struct encoding length
FieldConfig::new_struct(sort_options, estimated_length)
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
