milenkovicm commented on code in PR #1318:
URL:
https://github.com/apache/datafusion-ballista/pull/1318#discussion_r2348950076
##########
ballista/executor/src/flight_service.rs:
##########
@@ -185,16 +189,68 @@ impl FlightService for BallistaFlightService {
) -> Result<Response<Self::DoActionStream>, Status> {
let action = request.into_inner();
- let _action = decode_protobuf(&action.body).map_err(|e|
from_ballista_err(&e))?;
-
- Err(Status::unimplemented("do_action"))
+ match action.r#type.as_str() {
+ // Block transfer will transfer arrow ipc file block by block
+ // without decoding or decompressing, this will provide less
resource utilization
+ // as file are not decoded nor decompressed/compressed. Usually
this would transfer less data across
Review Comment:
just small note @alamb stream is compressed, shuffle file at rest (at local
directory) is already compressed. point i was trying to make is, when we have
compressed file, there is no need to uncompressed it just to be compressed
again. we can just ship file across, block by block where it will be
decompressed and decoded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
