westonpace commented on issue #15280: URL: https://github.com/apache/arrow/issues/15280#issuecomment-1379596541
Thanks for the detailed analysis and the suggestions. `engine` echos back to a time when we were thinking there might be potentially more formats like Substrait that could drive the engine. It was an early standin for "acero" as the name wasn't agreed at the time. I'm fine getting rid of the term "engine" and merging substrait & acero and agree that is a good idea. I'm a little hesitant about lumping "datasets" with Acero. Datasets is somewhat heavyweight, it requires parquet, orc, csv, filesystems, etc. None of those things are relevant to the core Acero. One could imagine Acero being used without any need for file I/O. For example, flight input, apply some compute operations, flight output. So my preference would be second chart you drew that has libacero and libacero_dataset. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
