rdettai commented on pull request #1010: URL: https://github.com/apache/arrow-datafusion/pull/1010#issuecomment-925611895
Thanks for all your great feedback! > How do you deal with a table with mixed file format? I didn't get the possibilities in the current implementation. I perceive it as the biggest benefit from this reorganization. Frankly speaking, I have never met a case of mixed file formats, so I wouldn't really know what is important to take into consideration. Can you describe your usecase precisely? Do you have an example of API that supports it? > I agree with @alamb on putting ObjectStore listing API in use in this PR, I think we have all prerequisites needed to use it for real. Since you already made scan in TableProvider async and doing the refactor of data sources in this one. Sold! I'm working in it 😄 > Will this refactor wipe out all the git history for each source file? both in datasource module and in physical_plan module? I think at least we should remove the deprecated ones in the same PR to preserve git history for each file? I think that the change is so massive that git will not recognize it as the previous files anyway, especially in the `datasource` module. This is a bit annoying but not sure we can do much about it :grimacing: > Again, does it make much sense to reorder the ones in physcial_plan? I didn't quite get the reason here. By "reorder", you mean moving them to a separate folder? The `physical_plan` folder is pretty huge, so I thought restructuring it a bit wouldn't harm. Also, it creates a nice symmetry with the folder structure in `datasource`. > I think at least we should remove the deprecated ones in the same PR to preserve git history for each file? We can, but it would make this PR huuuuuuuuge. From the development side, that's ok, but for the reviewers it might get tricky. @alamb @houqp what do you think about it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
