Yes, I think I'm following these instructions. And the file is not completely ignored. It creates additional format definitions. Let's say I white list some formats in my storage configuration and Drill adds more formats (which I don't want). Is there another way to start a "vanilla" Drill installation with my own configurations?
Stefan On Mon, Jul 10, 2023 at 6:17 PM Charles Givre <cgi...@gmail.com> wrote: > Hi Stefan, > My apologies.. Ok.. so the issue is that the storage-plugins-override.conf > is being ignored. I've never actually used this feature, so I wasn't > familiar with it, but are you folllowing the instructions here [1] with > respect to configuration and restarting Drill? My suggestion would be to > remove all the plugins in the UI and only specify them in the .conf file. > Drill has an order of precedence and I suspect what is happening is that > the UI versions have a higher priority than the .conf versions. Does that > make sense? > > -- C > > [1]: > https://drill.apache.org/docs/configuring-storage-plugins/#configuring-storage-plugins-with-the-storage-plugins-overrideconf-file > > > > > On Jul 10, 2023, at 12:06 PM, Stefan Ziegler < > stefan.ziegler...@gmail.com> wrote: > > > > Hi Charles > > > > I use a "storage-plugins-override.conf" file. My attempt is to have the > > configuration for my storages in a single file and Drill can pick up the > > configuration on startup. I put "storage-plugins-override.conf" in the > conf > > directory and Drill creates the storages on startup but (and that is my > > problem) also creates all formats for every storage defined in my config > > file. E.g. I have a (local) file type storage and I define two formats > > (parquet and json) in it. Drill does not respect my restriction to two > > formats in the config file but creates all formats known to Drill (like > > iceberg, xml etc.). > > > > regards > > Stefan > > > > On Mon, Jul 10, 2023 at 5:30 PM Charles Givre <cgi...@gmail.com> wrote: > > > >> HI Stefan, > >> Thanks for your interest in Drill. You have to define the format config > >> for each storage plugin. Otherwise Drill doesn't know what extension to > >> associate with what format plugin. Out of curiosity, why are you using > the > >> .conf files for this? > >> -- C > >> > >> > >>> On Jul 9, 2023, at 12:03 PM, Stefan Ziegler < > stefan.ziegler...@gmail.com> > >> wrote: > >>> > >>> Not defining a format seems to prevent the user from querying the > >> specific > >>> format. E.g. after deleting the xml format definition in the web gui, > I'm > >>> not able to query xml files anymore. So I guess my assumption was > right. > >>> > >>> Stefan > >>> > >>> On Sun, Jul 9, 2023 at 5:41 PM Stefan Ziegler < > >> stefan.ziegler...@gmail.com> > >>> wrote: > >>> > >>>> Btw: I assumed that the list of formats act as a restriction. Probably > >> I'm > >>>> wrong. > >>>> > >>>> Stefan > >>>> > >>>> On Sun, Jul 9, 2023 at 5:27 PM Stefan Ziegler < > >> stefan.ziegler...@gmail.com> > >>>> wrote: > >>>> > >>>>> Hi > >>>>> > >>>>> I'm using storage-plugins-override.conf to configure the storage > >> plugins > >>>>> on startup. My storage configurations contain only one or two formats > >>>>> (parquet, json, csv). Checking the storages in the web gui I noticed > >> that > >>>>> for all the storages all formats are enabled, e.g. msaccess, iceberg > >> etc. > >>>>> > >>>>> Is this on purpose or did I do something wrong? > >>>>> > >>>>> Example configuration: > >>>>> > >>>>> "storage": { > >>>>> dfs: { > >>>>> type: "file", > >>>>> connection: "file:///", > >>>>> workspaces: { > >>>>> "tmp": { > >>>>> "location": "/tmp", > >>>>> "writable": true, > >>>>> "defaultInputFormat": null, > >>>>> "allowAccessOutsideWorkspace": false > >>>>> }, > >>>>> "root": { > >>>>> "location": "/", > >>>>> "writable": false, > >>>>> "defaultInputFormat": null, > >>>>> "allowAccessOutsideWorkspace": false > >>>>> } > >>>>> }, > >>>>> formats: { > >>>>> "parquet": { > >>>>> "type": "parquet" > >>>>> }, > >>>>> "json": { > >>>>> "type": "json", > >>>>> "extensions": [ > >>>>> "json" > >>>>> ] > >>>>> } > >>>>> }, > >>>>> enabled: true > >>>>> } > >>>>> } > >>>>> > >>>>> regards > >>>>> Stefan > >>>>> > >>>> > >> > >> > >
