tokoko commented on issue #54603: URL: https://github.com/apache/spark/issues/54603#issuecomment-5807424082
@holdenk hey, I have a draft of the SPIP [document](https://github.com/tokoko/spark-adbc/blob/main/SPIP.md) in the repo. Should I turn it into a Google doc and we can collaborate there? I suppose a JIRA/formal proposal stuff can wait, but I can start on that as well if you think that's necessary. > I've got an idea on how we could do something fun here to allow for parallel direct reads from execs too. I had two "types" of read parallelism in mind: - Driver-driven partitioning via `AdbcStatement.executePartitions` (Driver here refers to adbc driver, not the spark driver) - Spark calls `executePartitions` during `planInputPartitions`, adbc driver hands back opaque partition descriptors, one per Spark task. - Client-driven partitioning - this would basically mirror jdbc data source with `partitionColumn` / `lowerBound` / `upperBound` / `numPartitions` options. There's no reason why they can't be used together as well. Is that along the lines of what you were thinking? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
