I think you should include that text verbatim, in the docs. Very helpful to developing an instinct for writing UDFs that work. And if I can understand it (and I do), then I would think most good Java devs will too.
On 7/20/15, 11:15 AM, "Jacques Nadeau" <[email protected]> wrote: >Hey Stefan, > >Can you propose some edits/updates to the documentation? The doc is >maintained on github [1]. > >The key thing to understand is that a UDF is slightly incomplete Java. >This is because Drill actually rips apart the functionality of the UDF and >recomposes it directly within expression evaluation code. This gives a >substantial performance and memory benefit but also creates some >challenges. As such, there are some key rules one should follow: > > - Don't use imports > - Both class file and source file have to be on the classpath (yes, Drill >uses the source of the function) > - Any JAR files holding UDF resources must include drill-module.conf >marker file so that Drill knows to include that JAR in consideration for >UDF loading > - ValueHolders should be treated as structs. As such, don't call any >methods on ValueHolders > >If you fork the documentation and provide a pull request, doc people >(Kristine and Bridget) will work to include your feedback so that others >don't find as many challenges as you did. > >thanks, >Jacques > >[1] >https://github.com/apache/drill/tree/gh-pages/_docs/develop-custom-functions > >On Mon, Jul 20, 2015 at 7:19 AM, Stefán Baxter <[email protected]> >wrote: > >> Hi, >> >> This is working now. >> >> This, unless someone corrects me, is the code needed to get a string >> parameter for a UDF: >> >> >> - input2.buffer.toString(input2.start, >> input2.end-input2.start,java.nio.charset.Charset.defaultCharset()) >> >> Regards, >> -Stefan (not the happiest camper) >> >> >> On Mon, Jul 20, 2015 at 2:07 PM, Stefán Baxter <[email protected]> >> wrote: >> >> > Hi, >> > >> > After going through the log this is clear what is happening (once the >> > Drill picked up the UDF a bit earlier this morning). >> > >> > I'm calling the VarCharHolder.toString() to get the text value for the >> > parameter and that is throwing this exception. >> > >> > Two observations: >> > >> > 1. Calling a deprecated function, even though that its not optimal, >> > usually does not cause such drastic results. >> > 2. As far as I can see there is no easy way to get the string value of >> > a property without carving out a piece of the buffer. >> > >> > What am I missing here? >> > >> > Regards, >> > -Stefan >> > >> > >> > On Mon, Jul 20, 2015 at 12:58 PM, Jacques Nadeau <[email protected]> >> > wrote: >> > >> >> Can you enable verbose errors at the session level? It may reveal more >> >> about what is failing. >> >> On Jul 20, 2015 5:32 AM, "Stefán Baxter" <[email protected]> >> >> wrote: >> >> >> >> > Hi Jim, >> >> > >> >> > I have made those changes and I'm wondering if you can runs this be >> >> using >> >> > the two .jar files that the "mvn package" places in the target >> >> directory? >> >> > >> >> > I have tried to have Drill pick those up but the error now is: >> >> > >> >> > Error: SYSTEM ERROR: UnsupportedOperationException >> >> > Fragment 0:0 >> >> > [Error Id: da589dd4-4cfd-4659-8b93-219074ab8c72 on localhost:31010] >> >> > (state=,code=0) >> >> > >> >> > It seems to indicate that it's picking up the functions but that they >> >> can >> >> > not be run. >> >> > >> >> > Regards, >> >> > - Stefán >> >> > >> >> > On Mon, Jul 20, 2015 at 1:25 AM, Jim Bates <[email protected]> >> wrote: >> >> > >> >> > > I pulled out your udf class and threw it into my package. It worked >> >> for >> >> > me >> >> > > with a few modifications. >> >> > > >> >> > > You can not have imported classes or method references in your eval >> or >> >> > > setup methods as the code will get pulled out and executed somewhere >> >> else >> >> > > and it won't be able to find it. With that in mind, Period will >> need >> >> to >> >> > be >> >> > > org.joda.time.Period, DateTime will need to be >> org.joda.time.DateTime >> >> and >> >> > > roundTimeStamp will need to be >> >> > > com.activitystream.drill.udfs.ASUserDefinedFunctions.roundTimeStamp. >> >> > > >> >> > > >> >> > > >> >> > > On Sun, Jul 19, 2015 at 7:02 PM, Stefán Baxter < >> >> > [email protected]> >> >> > > wrote: >> >> > > >> >> > > > Hi, >> >> > > > >> >> > > > The project can be found here: >> >> > > > https://github.com/acmeguy/asdrill >> >> > > > >> >> > > > Thank you, >> >> > > > -Stefán >> >> > > > >> >> > > > On Sun, Jul 19, 2015 at 11:57 PM, Stefán Baxter < >> >> > > [email protected] >> >> >> >> > > > > >> >> > > > wrote: >> >> > > > >> >> > > > > Hi, >> >> > > > > >> >> > > > > I'm more than happy to share the little that is there (I will >> >> publish >> >> > > it >> >> > > > > on github and send link tomorrow). >> >> > > > > >> >> > > > > I ended up copying my UDF (singl-file.java) into the >> >> > > > simple-drill-function >> >> > > > > project where it got picked up. >> >> > > > > >> >> > > > > Then I discovered a whole new set of dependencies/limitations >> >> > > > > >> >> > > > > - The UDF are recompiled - any imports are invalid or at >> least >> >> > > > > overwritten >> >> > > > > - import org.joda.time.Period; (means that Period class is >> not >> >> > > > > resolved on runtime) >> >> > > > > - Error: SYSTEM ERROR: CompileException: Line 71, Column 26: >> >> > Cannot >> >> > > > > determine simple type name "Period" >> >> > > > > >> >> > > > > - Calling any "outside" functions, like I was calling a >> static >> >> > > > > function of the new class (same file), leads to an errors as >> >> they >> >> > > are >> >> > > > not >> >> > > > > not resolved >> >> > > > > - Error: SYSTEM ERROR: CompileException: Line 70, Column >> 35: A >> >> > > method >> >> > > > > named "roundTimeStamp" is not declared in any enclosing class >> >> nor >> >> > > any >> >> > > > > supertype, nor through a static import >> >> > > > > >> >> > > > > Perhaps this was mentioned in the documentation but this is, at >> >> the >> >> > > very >> >> > > > > least, not straight forward and super-inviting. >> >> > > > > >> >> > > > > Thank you for your assistance, we will keep trying :) >> >> > > > > >> >> > > > > Regards, >> >> > > > > -Stefan >> >> > > > > >> >> > > > > >> >> > > > > On Sun, Jul 19, 2015 at 11:34 PM, Tugdual Grall < >> >> [email protected]> >> >> > > > wrote: >> >> > > > > >> >> > > > >> Hi Stefan, >> >> > > > >> >> >> > > > >> Do you think you can share your complete project ? >> >> > > > >> >> >> > > > >> This will help to debug it for you. >> >> > > > >> >> >> > > > >> T >> >> > > > >> >> >> > > > >> On Sunday, July 19, 2015, Stefán Baxter < >> >> [email protected]> >> >> > > > >> wrote: >> >> > > > >> >> >> > > > >> > Hi Ted, >> >> > > > >> > >> >> > > > >> > I fetched this, built it and deployed it without problems. >> >> > > > >> > I can not see any real difference other than this deploys two >> >> .jar >> >> > > (I >> >> > > > >> tried >> >> > > > >> > that as well earlier). >> >> > > > >> > >> >> > > > >> > I'm still trying to figure out why Drill is not picking up my >> >> UDFs >> >> > > > >> > >> >> > > > >> > Regards, >> >> > > > >> > -Stefán >> >> > > > >> > >> >> > > > >> > On Sun, Jul 19, 2015 at 10:45 PM, Ted Dunning < >> >> > > [email protected] >> >> > > > >> > <javascript:;>> wrote: >> >> > > > >> > >> >> > > > >> > > Stefan, >> >> > > > >> > > >> >> > > > >> > > Have you seen this github project: >> >> > > > >> > > >> >> > > > >> > > https://github.com/mapr-demos/simple-drill-functions >> >> > > > >> > > >> >> > > > >> > > ? >> >> > > > >> > > >> >> > > > >> > > >> >> > > > >> > > On Sun, Jul 19, 2015 at 2:14 PM, Stefán Baxter < >> >> > > > >> > [email protected] <javascript:;>> >> >> > > > >> > > wrote: >> >> > > > >> > > >> >> > > > >> > > > Hi Jim, >> >> > > > >> > > > >> >> > > > >> > > > I'm still not able to make this work. Do you have a >> sample >> >> > .jar >> >> > > > file >> >> > > > >> > > with a >> >> > > > >> > > > small example that you are running? >> >> > > > >> > > > >> >> > > > >> > > > Regards, >> >> > > > >> > > > -Stefan >> >> > > > >> > > > >> >> > > > >> > > > On Sun, Jul 19, 2015 at 6:46 PM, Ted Dunning < >> >> > > > [email protected] >> >> > > > >> > <javascript:;>> >> >> > > > >> > > > wrote: >> >> > > > >> > > > >> >> > > > >> > > > > >> >> > > > >> > > > > Sounds like a fine example, not because of >> sophistication >> >> > but >> >> > > > >> because >> >> > > > >> > > it >> >> > > > >> > > > > deals with dates. >> >> > > > >> > > > > >> >> > > > >> > > > > Check the drill logs. It is likely that drill is >> grumpy >> >> > about >> >> > > > >> > > something >> >> > > > >> > > > > in your udf or packaging. >> >> > > > >> > > > > >> >> > > > >> > > > > Also, feel free to snitch the pom from the simple >> >> examples >> >> > in >> >> > > > >> order >> >> > > > >> > to >> >> > > > >> > > > get >> >> > > > >> > > > > the pieces assembled and packaged correctly. >> >> > > > >> > > > > >> >> > > > >> > > > > Sent from my iPhone >> >> > > > >> > > > > >> >> > > > >> > > > > > On Jul 19, 2015, at 11:25, Stefán Baxter < >> >> > > > >> > [email protected] <javascript:;>> >> >> > > > >> > > > > wrote: >> >> > > > >> > > > > > >> >> > > > >> > > > > > Hi Jim, >> >> > > > >> > > > > > >> >> > > > >> > > > > > My UDF is currently so simple that I'm not sure you >> >> need >> >> > it >> >> > > > (or >> >> > > > >> > want >> >> > > > >> > > > it). >> >> > > > >> > > > > > >> >> > > > >> > > > > > It basically just rounds a timestamp value with ISO >> >> 8601 >> >> > > > periods >> >> > > > >> > > > > > asRoundTimestamp(timestampvalue,'PT10M'). >> >> > > > >> > > > > > >> >> > > > >> > > > > > I would be more than happy to contribute to your >> >> project >> >> > > > rather >> >> > > > >> > than >> >> > > > >> > > > > > building our own :). >> >> > > > >> > > > > > >> >> > > > >> > > > > > Is the repo public? >> >> > > > >> > > > > > >> >> > > > >> > > > > > Regards, >> >> > > > >> > > > > > -Stefan >> >> > > > >> > > > > > >> >> > > > >> > > > > > >> >> > > > >> > > > > > >> >> > > > >> > > > > >> On Sun, Jul 19, 2015 at 6:18 PM, Jim Bates < >> >> > > > >> [email protected] >> >> > > > >> > <javascript:;>> >> >> > > > >> > > > wrote: >> >> > > > >> > > > > >> >> >> > > > >> > > > > >> Maven will typically create a jar for class and a >> jar >> >> for >> >> > > > >> source >> >> > > > >> > > when >> >> > > > >> > > > > told >> >> > > > >> > > > > >> to do so. I just include the source files in the >> same >> >> jar >> >> > > as >> >> > > > >> the >> >> > > > >> > > class >> >> > > > >> > > > > >> files. There is a github example drill udf project >> we >> >> are >> >> > > > >> working >> >> > > > >> > on >> >> > > > >> > > > to >> >> > > > >> > > > > >> include several examples to simplify the learning >> >> curve. >> >> > If >> >> > > > >> your >> >> > > > >> > > > > >> interested... I'd love to have you add your udf. >> >> > > > >> > > > > >> On Jul 19, 2015 12:59 PM, "Stefán Baxter" < >> >> > > > >> > > [email protected] <javascript:;>> >> >> > > > >> > > > > >> wrote: >> >> > > > >> > > > > >> >> >> > > > >> > > > > >>> Hi again, >> >> > > > >> > > > > >>> >> >> > > > >> > > > > >>> Going over the documentation once more I came >> across >> >> > this: >> >> > > > >> > > > > >>> >> >> > > > >> > > > > >>> - Add the sources and classes JAR files to >> Drill’s >> >> > > > >> classpath. >> >> > > > >> > > > > >>> >> >> > > > >> > > > > >>> I'm only including a standard .jar (with classes no >> >> > > sources) >> >> > > > >> and >> >> > > > >> > I >> >> > > > >> > > > > wonder >> >> > > > >> > > > > >>> why this dependency is listed. >> >> > > > >> > > > > >>> >> >> > > > >> > > > > >>> Anyways, that it the next step. >> >> > > > >> > > > > >>> >> >> > > > >> > > > > >>> A sample .jar that works with the drill-config >> >> > counterpart >> >> > > > >> would >> >> > > > >> > > be a >> >> > > > >> > > > > >>> welcome addition to the documentation. >> >> > > > >> > > > > >>> >> >> > > > >> > > > > >>> Regards, >> >> > > > >> > > > > >>> -Stefan >> >> > > > >> > > > > >>> >> >> > > > >> > > > > >>> >> >> > > > >> > > > > >>> On Sun, Jul 19, 2015 at 5:53 PM, Stefán Baxter < >> >> > > > >> > > > > >> [email protected] <javascript:;>> >> >> > > > >> > > > > >>> wrote: >> >> > > > >> > > > > >>> >> >> > > > >> > > > > >>>> Hi Jim, >> >> > > > >> > > > > >>>> >> >> > > > >> > > > > >>>> Now I have added the file to the jar (both root >> and >> >> > > > resources >> >> > > > >> > > > folder) >> >> > > > >> > > > > >> but >> >> > > > >> > > > > >>>> that does not seem to change anything. >> >> > > > >> > > > > >>>> >> >> > > > >> > > > > >>>> Any additional ideas? >> >> > > > >> > > > > >>>> >> >> > > > >> > > > > >>>> Regards, >> >> > > > >> > > > > >>>> -Stefan >> >> > > > >> > > > > >>>> >> >> > > > >> > > > > >>>> On Sun, Jul 19, 2015 at 5:40 PM, Stefán Baxter < >> >> > > > >> > > > > >>> [email protected] <javascript:;>> >> >> > > > >> > > > > >>>> wrote: >> >> > > > >> > > > > >>>> >> >> > > > >> > > > > >>>>> NO! >> >> > > > >> > > > > >>>>> >> >> > > > >> > > > > >>>>> Thank you , will do that right now :) >> >> > > > >> > > > > >>>>> >> >> > > > >> > > > > >>>>> On Sun, Jul 19, 2015 at 5:38 PM, Jim Bates < >> >> > > > >> > [email protected] <javascript:;>> >> >> > > > >> > > > > >> wrote: >> >> > > > >> > > > > >>>>> >> >> > > > >> > > > > >>>>>> Did you include a file drill-module.conf in your >> >> jar >> >> > > > along >> >> > > > >> > with >> >> > > > >> > > > > >> source >> >> > > > >> > > > > >>>>>> files? >> >> > > > >> > > > > >>>>>> On Jul 19, 2015 12:20 PM, "Stefán Baxter" < >> >> > > > >> > > > > [email protected] <javascript:;> >> >> >> >> > > > >> >> >> > > > >> > > > > >>> >> >> > > > >> > > > > >>>>>> wrote: >> >> > > > >> > > > > >>>>>> >> >> > > > >> > > > > >>>>>>> Hi, >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> I'm trying to deploy a UDF that I have written >> >> > > according >> >> > > > >> to >> >> > > > >> > the >> >> > > > >> > > > > >>>>>>> documentation. >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> I have also: >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> 1. Copied the jar file >> >> > > > >> > > > > >>>>>>> to jars/3rdparty >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> 2. Changed the config >> >> "conf/drill-override.conf" >> >> > to >> >> > > > >> > include: >> >> > > > >> > > > > >>>>>>> drill.logical.function.package += >> >> > > > >> > > > > >> >> >> > > > >> > > >> >> > > > >> >> ["org.apache.drill.exec.expr.fn.impl","com.activitystream.drill.udfs"] >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> 3. Rerun the Drillbit >> >> > > > >> > > > > >>>>>>> bin/drillbit.sh start >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> The log shows no sign of the the UDF being >> loaded >> >> > but >> >> > > > if I >> >> > > > >> > use >> >> > > > >> > > it >> >> > > > >> > > > > >> in >> >> > > > >> > > > > >>> a >> >> > > > >> > > > > >>>>>> SQL >> >> > > > >> > > > > >>>>>>> statement then this error is thrown: >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> Error: PARSE ERROR: From line 1, column 184 to >> >> line >> >> > 1, >> >> > > > >> column >> >> > > > >> > > > 219: >> >> > > > >> > > > > >> No >> >> > > > >> > > > > >>>>>> match >> >> > > > >> > > > > >>>>>>> found for function signature >> >> asRoundTimeStamp(<ANY>, >> >> > > > >> > > <CHARACTER>) >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> The relevant part of the UDF source: >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> package com.activitystream.drill.udfs; >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> public class ASUserDefinedFunctions { >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> @FunctionTemplate(name = "asRoundTimeStamp", >> >> > scope >> >> > > = >> >> > > > >> > > > > >>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls = >> >> > > > >> > > > > >>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL) >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> public static class RoundTimeStamp >> implements >> >> > > > >> > > DrillSimpleFunc >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> Can anyone please assist? >> >> > > > >> > > > > >>>>>>> >> >> > > > >> > > > > >>>>>>> Regards, >> >> > > > >> > > > > >>>>>>> -Stefan >> >> > > > >> > > > > >> >> >> > > > >> > > > > >> >> > > > >> > > > >> >> > > > >> > > >> >> > > > >> > >> >> > > > >> >> >> > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> > >> > >>
