Let me see what I can do. Thanks. On Fri, May 31, 2019 at 2:17 PM Charles Givre <[email protected]> wrote:
> Hi Nicolas, > > You have two options: > 1. You can develop format plugins and UDFs in Drill by adding them to the > contrib/ folder and then test them with unit tests. Take a look at this PR > as an example[1]. If you're intending to submit your work to Drill for > inclusion, this would be my recommendation as you can write the unit tests > as you go, and it doesn't take very long to build and you can debug. > 2. Alternatively, you can package the code separately as shown here[2]. > However, this option requires you to build it, then copy the jars over to > DRILL_HOME/jars/3rd_party along with any dependencies, then run Drill. I'm > not sure how you could write unit tests this way. > > I hope this helps. > > > [1]: https://github.com/apache/drill/pull/1749 > [2]: https://github.com/cgivre/drill-excel-plugin > > > > On May 31, 2019, at 8:06 AM, Nicolas A Perez <[email protected]> > wrote: > > > > Paul, > > > > Is it possible to develop my plugin outside of the drill code, let's say > in > > my own repository and then package it and add it to the location where > the > > plugins live? Does that work, too? I just find annoying to deal with the > > full drill code in order to develop a plugin. At the same time, I might > > want to detach the development of plugins from the drill life cycle > itself. > > > > Please advise. > > > > Best Regards, > > > > Nicolas A Perez > > > > On Thu, May 30, 2019 at 9:58 PM Paul Rogers <[email protected]> > > wrote: > > > >> Hi Nicolas, > >> > >> A quick check of the code suggests that AbstractWriter is a > >> Json-serialized description of the physical plan. It represents the > >> information sent from the planner to the execution engine, and is > >> interpreted by the scan operator. That is, it is the "physical plan." > >> > >> The question is, how does the execution engine translate create the > actual > >> writer based on the physical plan? The only good example seems to be for > >> the FileSystemPlugin. That particular storage plugin is complicated by > the > >> additional layer of the format plugins. > >> > >> There is a bit of magic here. Briefly, Drill uses a BatchCreator to > create > >> your writer. It does so via some Java introspection magic. Drill looks > for > >> all subclases of BatchCreator, the uses the type of the second argument > to > >> the getBatch() method to find the correct class. This may mean that you > >> need to create one with MapRDBFormatPluginConfig as the type of the > second > >> argument. > >> > >> The getBatch() method then creates the CloseableRecordBatch > >> implementation. This is a full Drill operator, meaning it must handle > the > >> Volcano iterator protocol. Looks like you can perhaps use > WriterRecordBatch > >> as the writer operator itself. (See EasyWriterBatchCreator and follow > the > >> code to understand the plumbing.) > >> > >> You create a RecordWriter to do the actual work. AFAIK, MapRDB supports > >> JSON data model (at least in some form). If this is the version you are > >> working on, the fastest development path might just be to copy the > >> JsonRecordWriter, and replace the writes to JSON with writes to MapRDB. > At > >> least this gives you a place to start looking. > >> > >> > >> A more general solution would be to build the writer using some of the > >> recent additions to Drill such as the row set mechanisms for reading a > >> record batch. But, since copying the JSON approach provides a quick & > dirty > >> solution, perhaps that is good enough for this particular use case. > >> > >> > >> In our book, we recommend building each step one-by-one and doing a > quick > >> test to verify that each step works as you expect. If you create your > >> BatchCreator, but not the writer, things won't actually work, but you > can > >> set a breakpoint in the getBatch() method to verify the Drill did find > your > >> class. And so on. > >> > >> > >> Thanks, > >> - Paul > >> > >> > >> > >> On Thursday, May 30, 2019, 3:05:39 AM PDT, Nicolas A Perez < > >> [email protected]> wrote: > >> > >> Can anyone give me an overview of how to implement AbstractRecordWriter? > >> > >> What are the mechanics it follows, what should I do and so on? It will > very > >> helpful. > >> > >> Best Regards, > >> > >> Nicolas A Perez > >> -- > >> > >> > -------------------------------------------------------------------------------------------- > >> Sent by Nicolas A Perez from my GMAIL account. > >> > >> > -------------------------------------------------------------------------------------------- > >> > > > > > > > > -- > > > -------------------------------------------------------------------------------------------- > > Sent by Nicolas A Perez from my GMAIL account. > > > -------------------------------------------------------------------------------------------- > > -- -------------------------------------------------------------------------------------------- Sent by Nicolas A Perez from my GMAIL account. --------------------------------------------------------------------------------------------
