paul-rogers edited a comment on issue #1873: DRILL-6096: Provide mechanism to configure text writer configuration URL: https://github.com/apache/drill/pull/1873#issuecomment-544277734 General question: the idea of giving the user more control is a good one. I'm seeing this need in multiple of recent plugins. I wonder, however, if we should offer a more scalable solution? We currently have plugin, session, config and table function options all which control some aspect of reading and writing. The plugin and config options really want to be "bulk" or "slow moving" settings: set them once for a system or file format. Session options have two problems. First, they introduce client state: they must be set and reset around each query that needs them. This prevents load balancing and fail-over. (By contrast, Impala keeps options in the client and sends them to the server on each query, allowing unlimited load balancing via round-robin and allows transparent fail over.) Further, session options are awkward for many tools: in addition to running a query, the client must send `ALTER SESSION` statements which are somehow associated with that query. Easy for our unit tests, not so easy for most BI tools. Table options look like what we want, but they have a number of issues, including that, when used, they "erase" any options set in the format plugin. Then, consider usability. Suppose I set the options and write a text file. What do I have to do to query that file I just wrote? Configure a specialized format plugin? Use table options? Seems we really need a more stable, mature solution; we're past the time where we can get away with workarounds. Is there some other solution we could consider? Better table options? Some syntax that allows setting of options in the query? Some kind of view or schema that encapsulates these options so that they don't have to be provided on every query? I don't have a solid suggestion; I'm simply pointing out the ease-of-use issue.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
