ozankabak commented on code in PR #6049:
URL: https://github.com/apache/arrow-datafusion/pull/6049#discussion_r1179692890
##########
datafusion/core/src/physical_plan/memory.rs:
##########
@@ -223,15 +245,365 @@ impl RecordBatchStream for MemoryStream {
}
}
+/// Execution plan for writing record batches to an in-memory table.
+pub struct MemoryWriteExec {
Review Comment:
This is simply a baby step towards implementing things like `COPY TO`s,
`INSERT INTO`s and functionality like that. In practical cases, these things
will write their outputs through a `TableProvider` like a `ListingTable`. How
do we test what that will look like and iterate on such functionality?
Basically, this is the simplest writer that one can perform end-to-end
(read-transform-write) experiments with, its main intention is for anyone to
use as a basis/example for implementing their read/write operators for their
own `TableProvider`s. I hope it makes a little more sense now.
> If you guys feel strongly that this is the right and maintainable solution
going forward I will defer to your judgement, but from where I see it just adds
complexity / technical debt for no reason.
Thank you -- rest assured we will be maintaining and helping with related
work going forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
