Yes that's my problem. It is a format plugin and I want it included in dfs storage plugin..
I see there is a format-maprdb in the contrib folder. I will do the same format-msgpack. Thanks jc On Thu, Sep 27, 2018 at 7:25 AM, Arina Yelchiyeva < arina.yelchiy...@gmail.com> wrote: > Jean-Claude, > > this is a good question. Previously new formats were added into the exec > module and thus modifying bootstrap-storage-plugins.json was making sense. > Since now we are adding new format plugins into the contrib directory, this > raises the question on how to publish new format in dfs storage plugin. > In StoragePluginRegistryImpl if we find several > bootstrap-storage-plugins.json we load data from all of them but if there > is a duplicate between storage plugins (ex: dfs), the duplicate would be > ignored. > Basically, we need to implement new mechanism that would merge two storage > plugins formats. > > P.S. > 1. There is a difference between storage plugin and storage format, I > believe you have implemented the second. > 2. I suggest we put all formats under contrib/format/.., so they are not > confused with plugins. > > Kind regards, > Arina > > On Thu, Sep 27, 2018 at 12:54 PM Vitalii Diravka <vita...@apache.org> > wrote: > > > I'm not sure what you exactly mean. All *bootstrap-storage-plugins.json* > > are registered and stored in Persistent Store > > for the first fresh instantiating of Drill, see > > *StoragePluginRegistryImpl.loadBootstrapPlugins() > > *for details. > > > > I have noticed that you have a typo in the > > *bootstrap-storage-plugins.json *file > > name (boostrap -> *bootstrap*). > > > > Also you can use *storage-plugins-override.conf *for configuring plugins > > configs during start-up [1]. > > > > [1] > > > > https://drill.apache.org/docs/configuring-storage-plugins/# > configuring-storage-plugins-with-the-storage-plugins-override.conf-file > > > > > > On Wed, Sep 26, 2018 at 10:17 PM Jean-Claude Cote <jcc...@gmail.com> > > wrote: > > > > > I see how the cluster.makeDataDir call will setup the configuration > for a > > > MsgpackFormatConfig. I'll update my test cases to use it. > > > > > > However there is another question related to this. If I don't modify > the > > > boostrap-storage-plugins.json then when I launch the drill-embeded it > > does > > > not know of the MsgpackFormatConfig. I need to use the web console to > > edit > > > the storage plugins. > > > > > > Is there a better way? > > > Thanks > > > Jean-Claude > > > > > > > > > On Wed, Sep 26, 2018 at 1:17 PM, Arina Yelchiyeva < > > > arina.yelchiy...@gmail.com> wrote: > > > > > > > Taking into the account that your code is in the contrib module, > > > modifying > > > > boostrap-storage-plugins.json does not make any sense. > > > > If you need to add your own format in unit tests, as Vitalii pointed > > out, > > > > TestCsv is a good example for this. > > > > > > > > Kind regards, > > > > Arina > > > > > > > > On Wed, Sep 26, 2018 at 7:07 PM Jean-Claude Cote <jcc...@gmail.com> > > > wrote: > > > > > > > > > I found the cause of the problem I had. It was not due to the fact > > that > > > > the > > > > > classloader did not find the resource. It is that my new > FormatPlugin > > > was > > > > > not registered into the > > > > > > > drill\exec\java-exec\src\main\resources\bootstrap-storage-plugins.json > > > > > > > > > > cp: { > > > > > type: "file", > > > > > connection: "classpath:///", > > > > > formats: { > > > > > "msgpack" : { > > > > > type: "msgpack", > > > > > extensions: [ "mp" ] > > > > > }, > > > > > > > > > > So the resource was rejected. I've added this entry to the > > > > > boostrap-storage-plugins.json and now it works. > > > > > > > > > > Thanks for all your help. > > > > > Jean-Claude > > > > > > > > > > > > > > > > > > > > On Wed, Sep 26, 2018 at 7:18 AM, Arina Yelchiyeva < > > > > > arina.yelchiy...@gmail.com> wrote: > > > > > > > > > > > This can also help: > > > > > > > > > > > > 1. create TEST resources folder where you'll copy your data. > > > > > > 2. use dirTestWatcher to copy data to the root / tmp / custom > > > > directory. > > > > > > 3. query the data: > > > > > > > > > > > > @Test > > > > > > public void t() { > > > > > > > > dirTestWatcher.copyResourceToRoot(Paths.get("complex_1.parquet")); > > > > > > queryBuilder().sql("select * from > dfs.`root`.`complex_1.parquet` > > > > > > ").printCsv(); > > > > > > } > > > > > > > > > > > > > > > > > > On Wed, Sep 26, 2018 at 12:37 PM Vitalii Diravka < > > vita...@apache.org > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Jean-Claude > > > > > > > > > > > > > > BaseTestQuery is deprecated. Please use ClusterTest instead. > > > > > > > See TestCsv.java for example. > > > > > > > > > > > > > > You can find more info about Drill Cluster-Fixture-Framework > > here: > > > > > > > > > > https://github.com/paul-rogers/drill/wiki/Cluster-Fixture-Framework > > > > > > > > > > > > > > On Wed, Sep 26, 2018 at 12:00 AM Jean-Claude Cote < > > > jcc...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > I have writing a msgpack storage plugin from drill. > > > > > > > > https://github.com/jcmcote/drill/tree/master/contrib/ > > > > storage-msgpack > > > > > > > > > > > > > > > > I'm now trying to write test cases like > > > > > > > > > > > > > > > > testBuilder() > > > > > > > > .sqlQuery("select * from cp.`msgpack/testBasic.mp`") > > > > > > > > .ordered() > > > > > > > > .baselineColumns("a").baselineValues("1"). > > > > baselineValues("1") > > > > > > > > .baselineColumns("b").baselineValues("2"). > > > > baselineValues("2") > > > > > > > > .build().run(); > > > > > > > > > > > > > > > > However when I run the test case it says it cannot find the > > > > > > > > msgpack/testBasic.mp file. However it is in my > > src/test/resources > > > > > > folder. > > > > > > > > > > > > > > > > Should this work? I'm I going at it the right way? > > > > > > > > Thanks > > > > > > > > jc > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
