Hi Charles
I use a "storage-plugins-override.conf" file. My attempt is to have the
configuration for my storages in a single file and Drill can pick up the
configuration on startup. I put "storage-plugins-override.conf" in the conf
directory and Drill creates the storages on startup but (and that is my
problem) also creates all formats for every storage defined in my config
file. E.g. I have a (local) file type storage and I define two formats
(parquet and json) in it. Drill does not respect my restriction to two
formats in the config file but creates all formats known to Drill (like
iceberg, xml etc.).
regards
Stefan
On Mon, Jul 10, 2023 at 5:30 PM Charles Givre <cgi...@gmail.com> wrote:
> HI Stefan,
> Thanks for your interest in Drill. You have to define the format config
> for each storage plugin. Otherwise Drill doesn't know what extension to
> associate with what format plugin. Out of curiosity, why are you using the
> .conf files for this?
> -- C
>
>
> > On Jul 9, 2023, at 12:03 PM, Stefan Ziegler <stefan.ziegler...@gmail.com>
> wrote:
> >
> > Not defining a format seems to prevent the user from querying the
> specific
> > format. E.g. after deleting the xml format definition in the web gui, I'm
> > not able to query xml files anymore. So I guess my assumption was right.
> >
> > Stefan
> >
> > On Sun, Jul 9, 2023 at 5:41 PM Stefan Ziegler <
> stefan.ziegler...@gmail.com>
> > wrote:
> >
> >> Btw: I assumed that the list of formats act as a restriction. Probably
> I'm
> >> wrong.
> >>
> >> Stefan
> >>
> >> On Sun, Jul 9, 2023 at 5:27 PM Stefan Ziegler <
> stefan.ziegler...@gmail.com>
> >> wrote:
> >>
> >>> Hi
> >>>
> >>> I'm using storage-plugins-override.conf to configure the storage
> plugins
> >>> on startup. My storage configurations contain only one or two formats
> >>> (parquet, json, csv). Checking the storages in the web gui I noticed
> that
> >>> for all the storages all formats are enabled, e.g. msaccess, iceberg
> etc.
> >>>
> >>> Is this on purpose or did I do something wrong?
> >>>
> >>> Example configuration:
> >>>
> >>> "storage": {
> >>> dfs: {
> >>> type: "file",
> >>> connection: "file:///",
> >>> workspaces: {
> >>> "tmp": {
> >>> "location": "/tmp",
> >>> "writable": true,
> >>> "defaultInputFormat": null,
> >>> "allowAccessOutsideWorkspace": false
> >>> },
> >>> "root": {
> >>> "location": "/",
> >>> "writable": false,
> >>> "defaultInputFormat": null,
> >>> "allowAccessOutsideWorkspace": false
> >>> }
> >>> },
> >>> formats: {
> >>> "parquet": {
> >>> "type": "parquet"
> >>> },
> >>> "json": {
> >>> "type": "json",
> >>> "extensions": [
> >>> "json"
> >>> ]
> >>> }
> >>> },
> >>> enabled: true
> >>> }
> >>> }
> >>>
> >>> regards
> >>> Stefan
> >>>
> >>
>
>
