andygrove commented on issue #619: URL: https://github.com/apache/datafusion-ballista/issues/619#issuecomment-6062384516
This is done as of #2038. `restrict_plan_to_partitions` (`ballista/scheduler/src/state/task_builder.rs`) rewrites each task's plan before encoding, so `FileScanConfig` keeps only the task's file groups and `ShuffleReaderExec` keeps only the task's locations. On a 256-partition scan stage the encoded bytes summed across all tasks come to about the size of the full plan, not 256 times it. mingmwang's concern about encoding a different plan per task is real, but it is cheap in practice (see #2520). Closing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
