alamb commented on issue #15173: URL: https://github.com/apache/datafusion/issues/15173#issuecomment-2845807026
There is another request here that is related I think (basically "row number within the file"): - https://github.com/apache/datafusion/issues/15892 It seems to me what has happened is that the existing ListingTableProvider in DataFusion has to walk a fine balance between eing relatively "simple" and additional functionality like this while widely useful I think also could be implemented outside the core with the existing APIs. As we add more features to ListingTable it will look more and more like a full featured Table Format, but there are already many implementations of such formats (for example Iceberg, Delta, etc) Thus I think it might be good to propose we creating a `datafusion-contrib` project with make a more full featured "TableProvider" that adds new features like: 1. Additional metadata clumns 2. https://github.com/apache/datafusion/issues/15892 3. etc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
