ryankert01 commented on code in PR #1402:
URL: https://github.com/apache/mahout/pull/1402#discussion_r3412984445
##########
qdp/qdp-core/src/readers/arrow_ipc.rs:
##########
@@ -150,16 +150,12 @@ impl DataReader for ArrowIPCReader {
MahoutError::Io("Failed to downcast to
ListArray".to_string())
})?;
+ // Phase 1: find sample_size from non-null rows and
validate consistency.
for i in 0..list_array.len() {
- let value_array = list_array.value(i);
- let float_array = value_array
- .as_any()
- .downcast_ref::<Float64Array>()
- .ok_or_else(|| {
- MahoutError::Io("List values must be
Float64".to_string())
- })?;
-
- let current_size = float_array.len();
+ if list_array.is_null(i) {
Review Comment:
I want to ask about the context of a possible null might exists in a stream
of data? We often assume the data user sent are correct. We need a boundry of
what we should check and what we shouldn't check for user. (to strike a balance
between performance and early stop when handling error data)
this pr already do well actually~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
