andygrove opened a new issue, #2839: URL: https://github.com/apache/datafusion-ballista/issues/2839
Follow-up from #2721. The standalone scheduler (\`SessionContext::standalone\`, \`new_standalone_scheduler_from_state\`) builds its own \`SchedulerConfig\`. It takes the physical extension codec from the session config, but there is no way to give it a \`PartitionSliceableResolver\`. A plan with a custom node that holds data per partition can therefore run on standalone only when its stages are not split between tasks. For example, a \`UNION ALL\` over a custom scan, as an Iceberg integration (#2638) would produce, fails with no way to fix it. A custom node can only reach a task if the physical codec can serialize it, so whoever registers the codec also knows how to restrict the node. Options: - register the resolver on the `SessionConfig` next to the codec (e.g. `SessionConfigExt::with_ballista_partition_sliceable_resolver`), and have the standalone scheduler, and perhaps the scheduler binary's session builder, pick it up from there, or - let the standalone builder accept a resolver, or a whole `SchedulerConfig`. The first keeps the codec and the resolver in one place, so the two can't disagree. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
