I’d concur with Arina’s suggestion. I do think this would be useful and make
it easier to make plugins “pluggable”.
In the meantime, should we recommend that developers of format-plugins include
their plugins in the bootstrap-storage-plugins.json? I was thinking also that
we might want to have some guidelines for unit tests for format plugins. I’m
doing some work on the HTTPD format plugin and found some issues which cause it
to throw NPEs.
— C
> On Feb 5, 2019, at 06:40, Arina Yelchiyeva <arina.yelchiy...@gmail.com> wrote:
>
> Hi all,
>
> Before we were adding new formats / plugins into the exec module. Eventually
> we came up to the point that exec package size is growing and adding plugin
> and format contributions is better to separate out in the different module.
> Now we have contrib module where we add such contributions. Plugins are
> pluggable, there are added automatically by means of having drill-module.conf
> file which points to the scanning packages.
> Format plugins are using the same approach, the only problem is that they are
> not added into bootstrap-storage-plugins.json. So when adding new format
> plugin, in order for it to automatically appear in Drill Web UI, developer
> has to update bootstrap file which is in the exec module.
> My suggestion we implement some functionality that would merge format config
> with the bootstrap one. For example, each plugin would have to have
> bootstrap-format.json file with the information to which plugin format should
> be added (structure the same as in bootstrap-storage-plugins.json):
> Example:
>
> {
> "storage":{
> dfs: {
> formats: {
> "psv" : {
> type: "msgpack",
> extensions: [ "mp" ]
> }
> }
> }
> }
> }
>
> Then during Drill start up such bootstrap-format.json files will be merged
> with bootstrap-storage-plugins.json.
>
>
> Current open PR for adding new format plugins:
> Format plugin for LTSV files - https://github.com/apache/drill/pull/1627
> SYSLOG (RFC-5424) Format Plugin - https://github.com/apache/drill/pull/1530
> Msgpack format reader - https://github.com/apache/drill/pull/1500
>
> Any suggestions?
>
> Kind regards,
> Arina
