timsaucer opened a new pull request, #103: URL: https://github.com/apache/datafusion-java/pull/103
**DO NOT MERGE. THIS IS A WORK IN PROGRESS.** ## Which issue does this PR close? None, but willing to open one for discussion. ## Rationale for this change This PR provides two major pieces - Support for Rust backed Table Providers to interact with Java session context - Exposing DataFusion table providers as Spark data sources With these you can use DataFusion designed table providers with existing spark workflows as a columnar data source. There may be additional opportunity with Comet to identify these tables and use their underlying implementations directly. This extends the usefulness of DataFusion backed sources. ## What changes are included in this PR? - Addition of a new crate - Addition of new java code to support FFI table providers - DataFusion data source for spark library that includes schema widening and type casting for types not supported by Spark - Full end to end example using an in memory datafusion table with pyspark ## Are these changes tested? - End to end test is included - TODO: Evaluate what other testing requirements are needed ## Are there any user-facing changes? There are user facing changes, but they are addition only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
