0lai0 commented on code in PR #1402:
URL: https://github.com/apache/mahout/pull/1402#discussion_r3414099806
##########
qdp/qdp-core/src/readers/arrow_ipc.rs:
##########
@@ -150,16 +150,12 @@ impl DataReader for ArrowIPCReader {
MahoutError::Io("Failed to downcast to
ListArray".to_string())
})?;
+ // Phase 1: find sample_size from non-null rows and
validate consistency.
for i in 0..list_array.len() {
- let value_array = list_array.value(i);
- let float_array = value_array
- .as_any()
- .downcast_ref::<Float64Array>()
- .ok_or_else(|| {
- MahoutError::Io("List values must be
Float64".to_string())
- })?;
-
- let current_size = float_array.len();
+ if list_array.is_null(i) {
Review Comment:
Thanks @ryankert01.
Good point! Arrow's List field defaults to nullable=true in most export
tools (pandas, polars, Parquet writers), so null outer rows can appear even
without user error.
The NullHandling policy lets the caller decide strictness. Reject for tight
pipelines, FillZero for tolerant ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
