timsaucer commented on issue #17713:
URL: https://github.com/apache/datafusion/issues/17713#issuecomment-3398250183
I've been digging into this a little today and I keep expanding the scope of
a refactor.
There is some easy stuff like switching from `SessionContext` to
`TaskContext`. The place we have `table_factory` doesn't *use* the factory
other than a check for supported file types. That, I think, is a nice to have
but not strictly necessary check. I could be wrong on that one.
What I came across next was that we are using a fair number of other things
in `datafusion/core` that we would have to refactor to get into other crates.
This means creating a new crate `datasource-arrow` for the Arrow IPC formatted
files. Next up was moving things like `ListingTable` over to `datasource` or
some other crate.
In my current intermediate state I need to pull in all of these crates
```toml
datafusion-catalog = { workspace = true }
datafusion-common = { workspace = true }
datafusion-datasource = { workspace = true }
datafusion-datasource-arrow = { workspace = true }
datafusion-datasource-csv = { workspace = true }
datafusion-datasource-json = { workspace = true }
datafusion-datasource-parquet = { workspace = true }
datafusion-execution = { workspace = true }
datafusion-expr = { workspace = true }
datafusion-functions-table = { workspace = true }
datafusion-physical-expr = { workspace = true }
datafusion-physical-plan = { workspace = true }
datafusion-proto-common = { workspace = true }
```
It's not undo-able but I find this refactor exploding in scope. I'm sure it
can be broken into parts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
