Hey Stefan, Can you propose some edits/updates to the documentation? The doc is maintained on github [1].
The key thing to understand is that a UDF is slightly incomplete Java. This is because Drill actually rips apart the functionality of the UDF and recomposes it directly within expression evaluation code. This gives a substantial performance and memory benefit but also creates some challenges. As such, there are some key rules one should follow: - Don't use imports - Both class file and source file have to be on the classpath (yes, Drill uses the source of the function) - Any JAR files holding UDF resources must include drill-module.conf marker file so that Drill knows to include that JAR in consideration for UDF loading - ValueHolders should be treated as structs. As such, don't call any methods on ValueHolders If you fork the documentation and provide a pull request, doc people (Kristine and Bridget) will work to include your feedback so that others don't find as many challenges as you did. thanks, Jacques [1] https://github.com/apache/drill/tree/gh-pages/_docs/develop-custom-functions On Mon, Jul 20, 2015 at 7:19 AM, Stefán Baxter <[email protected]> wrote: > Hi, > > This is working now. > > This, unless someone corrects me, is the code needed to get a string > parameter for a UDF: > > > - input2.buffer.toString(input2.start, > input2.end-input2.start,java.nio.charset.Charset.defaultCharset()) > > Regards, > -Stefan (not the happiest camper) > > > On Mon, Jul 20, 2015 at 2:07 PM, Stefán Baxter <[email protected]> > wrote: > > > Hi, > > > > After going through the log this is clear what is happening (once the > > Drill picked up the UDF a bit earlier this morning). > > > > I'm calling the VarCharHolder.toString() to get the text value for the > > parameter and that is throwing this exception. > > > > Two observations: > > > > 1. Calling a deprecated function, even though that its not optimal, > > usually does not cause such drastic results. > > 2. As far as I can see there is no easy way to get the string value of > > a property without carving out a piece of the buffer. > > > > What am I missing here? > > > > Regards, > > -Stefan > > > > > > On Mon, Jul 20, 2015 at 12:58 PM, Jacques Nadeau <[email protected]> > > wrote: > > > >> Can you enable verbose errors at the session level? It may reveal more > >> about what is failing. > >> On Jul 20, 2015 5:32 AM, "Stefán Baxter" <[email protected]> > >> wrote: > >> > >> > Hi Jim, > >> > > >> > I have made those changes and I'm wondering if you can runs this be > >> using > >> > the two .jar files that the "mvn package" places in the target > >> directory? > >> > > >> > I have tried to have Drill pick those up but the error now is: > >> > > >> > Error: SYSTEM ERROR: UnsupportedOperationException > >> > Fragment 0:0 > >> > [Error Id: da589dd4-4cfd-4659-8b93-219074ab8c72 on localhost:31010] > >> > (state=,code=0) > >> > > >> > It seems to indicate that it's picking up the functions but that they > >> can > >> > not be run. > >> > > >> > Regards, > >> > - Stefán > >> > > >> > On Mon, Jul 20, 2015 at 1:25 AM, Jim Bates <[email protected]> > wrote: > >> > > >> > > I pulled out your udf class and threw it into my package. It worked > >> for > >> > me > >> > > with a few modifications. > >> > > > >> > > You can not have imported classes or method references in your eval > or > >> > > setup methods as the code will get pulled out and executed somewhere > >> else > >> > > and it won't be able to find it. With that in mind, Period will > need > >> to > >> > be > >> > > org.joda.time.Period, DateTime will need to be > org.joda.time.DateTime > >> and > >> > > roundTimeStamp will need to be > >> > > com.activitystream.drill.udfs.ASUserDefinedFunctions.roundTimeStamp. > >> > > > >> > > > >> > > > >> > > On Sun, Jul 19, 2015 at 7:02 PM, Stefán Baxter < > >> > [email protected]> > >> > > wrote: > >> > > > >> > > > Hi, > >> > > > > >> > > > The project can be found here: > >> > > > https://github.com/acmeguy/asdrill > >> > > > > >> > > > Thank you, > >> > > > -Stefán > >> > > > > >> > > > On Sun, Jul 19, 2015 at 11:57 PM, Stefán Baxter < > >> > > [email protected] > >> > >> > > > > > >> > > > wrote: > >> > > > > >> > > > > Hi, > >> > > > > > >> > > > > I'm more than happy to share the little that is there (I will > >> publish > >> > > it > >> > > > > on github and send link tomorrow). > >> > > > > > >> > > > > I ended up copying my UDF (singl-file.java) into the > >> > > > simple-drill-function > >> > > > > project where it got picked up. > >> > > > > > >> > > > > Then I discovered a whole new set of dependencies/limitations > >> > > > > > >> > > > > - The UDF are recompiled - any imports are invalid or at > least > >> > > > > overwritten > >> > > > > - import org.joda.time.Period; (means that Period class is > not > >> > > > > resolved on runtime) > >> > > > > - Error: SYSTEM ERROR: CompileException: Line 71, Column 26: > >> > Cannot > >> > > > > determine simple type name "Period" > >> > > > > > >> > > > > - Calling any "outside" functions, like I was calling a > static > >> > > > > function of the new class (same file), leads to an errors as > >> they > >> > > are > >> > > > not > >> > > > > not resolved > >> > > > > - Error: SYSTEM ERROR: CompileException: Line 70, Column > 35: A > >> > > method > >> > > > > named "roundTimeStamp" is not declared in any enclosing class > >> nor > >> > > any > >> > > > > supertype, nor through a static import > >> > > > > > >> > > > > Perhaps this was mentioned in the documentation but this is, at > >> the > >> > > very > >> > > > > least, not straight forward and super-inviting. > >> > > > > > >> > > > > Thank you for your assistance, we will keep trying :) > >> > > > > > >> > > > > Regards, > >> > > > > -Stefan > >> > > > > > >> > > > > > >> > > > > On Sun, Jul 19, 2015 at 11:34 PM, Tugdual Grall < > >> [email protected]> > >> > > > wrote: > >> > > > > > >> > > > >> Hi Stefan, > >> > > > >> > >> > > > >> Do you think you can share your complete project ? > >> > > > >> > >> > > > >> This will help to debug it for you. > >> > > > >> > >> > > > >> T > >> > > > >> > >> > > > >> On Sunday, July 19, 2015, Stefán Baxter < > >> [email protected]> > >> > > > >> wrote: > >> > > > >> > >> > > > >> > Hi Ted, > >> > > > >> > > >> > > > >> > I fetched this, built it and deployed it without problems. > >> > > > >> > I can not see any real difference other than this deploys two > >> .jar > >> > > (I > >> > > > >> tried > >> > > > >> > that as well earlier). > >> > > > >> > > >> > > > >> > I'm still trying to figure out why Drill is not picking up my > >> UDFs > >> > > > >> > > >> > > > >> > Regards, > >> > > > >> > -Stefán > >> > > > >> > > >> > > > >> > On Sun, Jul 19, 2015 at 10:45 PM, Ted Dunning < > >> > > [email protected] > >> > > > >> > <javascript:;>> wrote: > >> > > > >> > > >> > > > >> > > Stefan, > >> > > > >> > > > >> > > > >> > > Have you seen this github project: > >> > > > >> > > > >> > > > >> > > https://github.com/mapr-demos/simple-drill-functions > >> > > > >> > > > >> > > > >> > > ? > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > On Sun, Jul 19, 2015 at 2:14 PM, Stefán Baxter < > >> > > > >> > [email protected] <javascript:;>> > >> > > > >> > > wrote: > >> > > > >> > > > >> > > > >> > > > Hi Jim, > >> > > > >> > > > > >> > > > >> > > > I'm still not able to make this work. Do you have a > sample > >> > .jar > >> > > > file > >> > > > >> > > with a > >> > > > >> > > > small example that you are running? > >> > > > >> > > > > >> > > > >> > > > Regards, > >> > > > >> > > > -Stefan > >> > > > >> > > > > >> > > > >> > > > On Sun, Jul 19, 2015 at 6:46 PM, Ted Dunning < > >> > > > [email protected] > >> > > > >> > <javascript:;>> > >> > > > >> > > > wrote: > >> > > > >> > > > > >> > > > >> > > > > > >> > > > >> > > > > Sounds like a fine example, not because of > sophistication > >> > but > >> > > > >> because > >> > > > >> > > it > >> > > > >> > > > > deals with dates. > >> > > > >> > > > > > >> > > > >> > > > > Check the drill logs. It is likely that drill is > grumpy > >> > about > >> > > > >> > > something > >> > > > >> > > > > in your udf or packaging. > >> > > > >> > > > > > >> > > > >> > > > > Also, feel free to snitch the pom from the simple > >> examples > >> > in > >> > > > >> order > >> > > > >> > to > >> > > > >> > > > get > >> > > > >> > > > > the pieces assembled and packaged correctly. > >> > > > >> > > > > > >> > > > >> > > > > Sent from my iPhone > >> > > > >> > > > > > >> > > > >> > > > > > On Jul 19, 2015, at 11:25, Stefán Baxter < > >> > > > >> > [email protected] <javascript:;>> > >> > > > >> > > > > wrote: > >> > > > >> > > > > > > >> > > > >> > > > > > Hi Jim, > >> > > > >> > > > > > > >> > > > >> > > > > > My UDF is currently so simple that I'm not sure you > >> need > >> > it > >> > > > (or > >> > > > >> > want > >> > > > >> > > > it). > >> > > > >> > > > > > > >> > > > >> > > > > > It basically just rounds a timestamp value with ISO > >> 8601 > >> > > > periods > >> > > > >> > > > > > asRoundTimestamp(timestampvalue,'PT10M'). > >> > > > >> > > > > > > >> > > > >> > > > > > I would be more than happy to contribute to your > >> project > >> > > > rather > >> > > > >> > than > >> > > > >> > > > > > building our own :). > >> > > > >> > > > > > > >> > > > >> > > > > > Is the repo public? > >> > > > >> > > > > > > >> > > > >> > > > > > Regards, > >> > > > >> > > > > > -Stefan > >> > > > >> > > > > > > >> > > > >> > > > > > > >> > > > >> > > > > > > >> > > > >> > > > > >> On Sun, Jul 19, 2015 at 6:18 PM, Jim Bates < > >> > > > >> [email protected] > >> > > > >> > <javascript:;>> > >> > > > >> > > > wrote: > >> > > > >> > > > > >> > >> > > > >> > > > > >> Maven will typically create a jar for class and a > jar > >> for > >> > > > >> source > >> > > > >> > > when > >> > > > >> > > > > told > >> > > > >> > > > > >> to do so. I just include the source files in the > same > >> jar > >> > > as > >> > > > >> the > >> > > > >> > > class > >> > > > >> > > > > >> files. There is a github example drill udf project > we > >> are > >> > > > >> working > >> > > > >> > on > >> > > > >> > > > to > >> > > > >> > > > > >> include several examples to simplify the learning > >> curve. > >> > If > >> > > > >> your > >> > > > >> > > > > >> interested... I'd love to have you add your udf. > >> > > > >> > > > > >> On Jul 19, 2015 12:59 PM, "Stefán Baxter" < > >> > > > >> > > [email protected] <javascript:;>> > >> > > > >> > > > > >> wrote: > >> > > > >> > > > > >> > >> > > > >> > > > > >>> Hi again, > >> > > > >> > > > > >>> > >> > > > >> > > > > >>> Going over the documentation once more I came > across > >> > this: > >> > > > >> > > > > >>> > >> > > > >> > > > > >>> - Add the sources and classes JAR files to > Drill’s > >> > > > >> classpath. > >> > > > >> > > > > >>> > >> > > > >> > > > > >>> I'm only including a standard .jar (with classes no > >> > > sources) > >> > > > >> and > >> > > > >> > I > >> > > > >> > > > > wonder > >> > > > >> > > > > >>> why this dependency is listed. > >> > > > >> > > > > >>> > >> > > > >> > > > > >>> Anyways, that it the next step. > >> > > > >> > > > > >>> > >> > > > >> > > > > >>> A sample .jar that works with the drill-config > >> > counterpart > >> > > > >> would > >> > > > >> > > be a > >> > > > >> > > > > >>> welcome addition to the documentation. > >> > > > >> > > > > >>> > >> > > > >> > > > > >>> Regards, > >> > > > >> > > > > >>> -Stefan > >> > > > >> > > > > >>> > >> > > > >> > > > > >>> > >> > > > >> > > > > >>> On Sun, Jul 19, 2015 at 5:53 PM, Stefán Baxter < > >> > > > >> > > > > >> [email protected] <javascript:;>> > >> > > > >> > > > > >>> wrote: > >> > > > >> > > > > >>> > >> > > > >> > > > > >>>> Hi Jim, > >> > > > >> > > > > >>>> > >> > > > >> > > > > >>>> Now I have added the file to the jar (both root > and > >> > > > resources > >> > > > >> > > > folder) > >> > > > >> > > > > >> but > >> > > > >> > > > > >>>> that does not seem to change anything. > >> > > > >> > > > > >>>> > >> > > > >> > > > > >>>> Any additional ideas? > >> > > > >> > > > > >>>> > >> > > > >> > > > > >>>> Regards, > >> > > > >> > > > > >>>> -Stefan > >> > > > >> > > > > >>>> > >> > > > >> > > > > >>>> On Sun, Jul 19, 2015 at 5:40 PM, Stefán Baxter < > >> > > > >> > > > > >>> [email protected] <javascript:;>> > >> > > > >> > > > > >>>> wrote: > >> > > > >> > > > > >>>> > >> > > > >> > > > > >>>>> NO! > >> > > > >> > > > > >>>>> > >> > > > >> > > > > >>>>> Thank you , will do that right now :) > >> > > > >> > > > > >>>>> > >> > > > >> > > > > >>>>> On Sun, Jul 19, 2015 at 5:38 PM, Jim Bates < > >> > > > >> > [email protected] <javascript:;>> > >> > > > >> > > > > >> wrote: > >> > > > >> > > > > >>>>> > >> > > > >> > > > > >>>>>> Did you include a file drill-module.conf in your > >> jar > >> > > > along > >> > > > >> > with > >> > > > >> > > > > >> source > >> > > > >> > > > > >>>>>> files? > >> > > > >> > > > > >>>>>> On Jul 19, 2015 12:20 PM, "Stefán Baxter" < > >> > > > >> > > > > [email protected] <javascript:;> > >> > >> > > > >> > >> > > > >> > > > > >>> > >> > > > >> > > > > >>>>>> wrote: > >> > > > >> > > > > >>>>>> > >> > > > >> > > > > >>>>>>> Hi, > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> I'm trying to deploy a UDF that I have written > >> > > according > >> > > > >> to > >> > > > >> > the > >> > > > >> > > > > >>>>>>> documentation. > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> I have also: > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> 1. Copied the jar file > >> > > > >> > > > > >>>>>>> to jars/3rdparty > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> 2. Changed the config > >> "conf/drill-override.conf" > >> > to > >> > > > >> > include: > >> > > > >> > > > > >>>>>>> drill.logical.function.package += > >> > > > >> > > > > >> > >> > > > >> > > > >> > > > > >> ["org.apache.drill.exec.expr.fn.impl","com.activitystream.drill.udfs"] > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> 3. Rerun the Drillbit > >> > > > >> > > > > >>>>>>> bin/drillbit.sh start > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> The log shows no sign of the the UDF being > loaded > >> > but > >> > > > if I > >> > > > >> > use > >> > > > >> > > it > >> > > > >> > > > > >> in > >> > > > >> > > > > >>> a > >> > > > >> > > > > >>>>>> SQL > >> > > > >> > > > > >>>>>>> statement then this error is thrown: > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> Error: PARSE ERROR: From line 1, column 184 to > >> line > >> > 1, > >> > > > >> column > >> > > > >> > > > 219: > >> > > > >> > > > > >> No > >> > > > >> > > > > >>>>>> match > >> > > > >> > > > > >>>>>>> found for function signature > >> asRoundTimeStamp(<ANY>, > >> > > > >> > > <CHARACTER>) > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> The relevant part of the UDF source: > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> package com.activitystream.drill.udfs; > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> public class ASUserDefinedFunctions { > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> @FunctionTemplate(name = "asRoundTimeStamp", > >> > scope > >> > > = > >> > > > >> > > > > >>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls = > >> > > > >> > > > > >>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL) > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> public static class RoundTimeStamp > implements > >> > > > >> > > DrillSimpleFunc > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> Can anyone please assist? > >> > > > >> > > > > >>>>>>> > >> > > > >> > > > > >>>>>>> Regards, > >> > > > >> > > > > >>>>>>> -Stefan > >> > > > >> > > > > >> > >> > > > >> > > > > > >> > > > >> > > > > >> > > > >> > > > >> > > > >> > > >> > > > >> > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
