Hi Nicolas, You have two options: 1. You can develop format plugins and UDFs in Drill by adding them to the contrib/ folder and then test them with unit tests. Take a look at this PR as an example[1]. If you're intending to submit your work to Drill for inclusion, this would be my recommendation as you can write the unit tests as you go, and it doesn't take very long to build and you can debug. 2. Alternatively, you can package the code separately as shown here[2]. However, this option requires you to build it, then copy the jars over to DRILL_HOME/jars/3rd_party along with any dependencies, then run Drill. I'm not sure how you could write unit tests this way.
I hope this helps. [1]: https://github.com/apache/drill/pull/1749 [2]: https://github.com/cgivre/drill-excel-plugin > On May 31, 2019, at 8:06 AM, Nicolas A Perez <[email protected]> wrote: > > Paul, > > Is it possible to develop my plugin outside of the drill code, let's say in > my own repository and then package it and add it to the location where the > plugins live? Does that work, too? I just find annoying to deal with the > full drill code in order to develop a plugin. At the same time, I might > want to detach the development of plugins from the drill life cycle itself. > > Please advise. > > Best Regards, > > Nicolas A Perez > > On Thu, May 30, 2019 at 9:58 PM Paul Rogers <[email protected]> > wrote: > >> Hi Nicolas, >> >> A quick check of the code suggests that AbstractWriter is a >> Json-serialized description of the physical plan. It represents the >> information sent from the planner to the execution engine, and is >> interpreted by the scan operator. That is, it is the "physical plan." >> >> The question is, how does the execution engine translate create the actual >> writer based on the physical plan? The only good example seems to be for >> the FileSystemPlugin. That particular storage plugin is complicated by the >> additional layer of the format plugins. >> >> There is a bit of magic here. Briefly, Drill uses a BatchCreator to create >> your writer. It does so via some Java introspection magic. Drill looks for >> all subclases of BatchCreator, the uses the type of the second argument to >> the getBatch() method to find the correct class. This may mean that you >> need to create one with MapRDBFormatPluginConfig as the type of the second >> argument. >> >> The getBatch() method then creates the CloseableRecordBatch >> implementation. This is a full Drill operator, meaning it must handle the >> Volcano iterator protocol. Looks like you can perhaps use WriterRecordBatch >> as the writer operator itself. (See EasyWriterBatchCreator and follow the >> code to understand the plumbing.) >> >> You create a RecordWriter to do the actual work. AFAIK, MapRDB supports >> JSON data model (at least in some form). If this is the version you are >> working on, the fastest development path might just be to copy the >> JsonRecordWriter, and replace the writes to JSON with writes to MapRDB. At >> least this gives you a place to start looking. >> >> >> A more general solution would be to build the writer using some of the >> recent additions to Drill such as the row set mechanisms for reading a >> record batch. But, since copying the JSON approach provides a quick & dirty >> solution, perhaps that is good enough for this particular use case. >> >> >> In our book, we recommend building each step one-by-one and doing a quick >> test to verify that each step works as you expect. If you create your >> BatchCreator, but not the writer, things won't actually work, but you can >> set a breakpoint in the getBatch() method to verify the Drill did find your >> class. And so on. >> >> >> Thanks, >> - Paul >> >> >> >> On Thursday, May 30, 2019, 3:05:39 AM PDT, Nicolas A Perez < >> [email protected]> wrote: >> >> Can anyone give me an overview of how to implement AbstractRecordWriter? >> >> What are the mechanics it follows, what should I do and so on? It will very >> helpful. >> >> Best Regards, >> >> Nicolas A Perez >> -- >> >> -------------------------------------------------------------------------------------------- >> Sent by Nicolas A Perez from my GMAIL account. >> >> -------------------------------------------------------------------------------------------- >> > > > > -- > -------------------------------------------------------------------------------------------- > Sent by Nicolas A Perez from my GMAIL account. > --------------------------------------------------------------------------------------------
