Hi, Just wanted to thank those that helped.
While I'm happy that the UDF is running I feel like it could have taken a lost shorter. I will contribute to the documentation but here is my main takeaway: - Source code needs to be included in the jar - it's used when the drillbits/queries are built/orchistrated - Always use full class qualifiers in the Eval() segment of your UDF - Don't forget to add the drill-module.conf to the resources folder of your project (should end up in th root of the jar) - Adding your udf package to drill-override.conf does not seem to matter - just copy the jar(s) with the .class and .java files to the jars/3rdparty directory - Feel free to user this code anyway you wish: - https://github.com/acmeguy/asdrill - Know about this project: - https://github.com/mapr-demos/simple-drill-functions Regards, -Stefán On Mon, Jul 20, 2015 at 2:23 PM, Jim Bates <[email protected]> wrote: > That's an apt description. The Holders that have a fixed size value objects > are simple to get at but the ones that have variable length objects are > pulled via the buffer. > > You can also use the StringFunctionHelpers. > > > org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(in > .start, in.end, in.buffer) > > Where 'in' is > > @Param NullableVarCharHolder in > > or > > @Param VarCharHolder in > > On Mon, Jul 20, 2015 at 9:07 AM, Stefán Baxter <[email protected]> > wrote: > > > Hi, > > > > After going through the log this is clear what is happening (once the > Drill > > picked up the UDF a bit earlier this morning). > > > > I'm calling the VarCharHolder.toString() to get the text value for the > > parameter and that is throwing this exception. > > > > Two observations: > > > > 1. Calling a deprecated function, even though that its not optimal, > > usually does not cause such drastic results. > > 2. As far as I can see there is no easy way to get the string value > of a > > property without carving out a piece of the buffer. > > > > What am I missing here? > > > > Regards, > > -Stefan > > > > > > On Mon, Jul 20, 2015 at 12:58 PM, Jacques Nadeau <[email protected]> > > wrote: > > > > > Can you enable verbose errors at the session level? It may reveal more > > > about what is failing. > > > On Jul 20, 2015 5:32 AM, "Stefán Baxter" <[email protected]> > > > wrote: > > > > > > > Hi Jim, > > > > > > > > I have made those changes and I'm wondering if you can runs this be > > using > > > > the two .jar files that the "mvn package" places in the target > > directory? > > > > > > > > I have tried to have Drill pick those up but the error now is: > > > > > > > > Error: SYSTEM ERROR: UnsupportedOperationException > > > > Fragment 0:0 > > > > [Error Id: da589dd4-4cfd-4659-8b93-219074ab8c72 on localhost:31010] > > > > (state=,code=0) > > > > > > > > It seems to indicate that it's picking up the functions but that they > > can > > > > not be run. > > > > > > > > Regards, > > > > - Stefán > > > > > > > > On Mon, Jul 20, 2015 at 1:25 AM, Jim Bates <[email protected]> > > wrote: > > > > > > > > > I pulled out your udf class and threw it into my package. It worked > > for > > > > me > > > > > with a few modifications. > > > > > > > > > > You can not have imported classes or method references in your eval > > or > > > > > setup methods as the code will get pulled out and executed > somewhere > > > else > > > > > and it won't be able to find it. With that in mind, Period will > need > > > to > > > > be > > > > > org.joda.time.Period, DateTime will need to be > org.joda.time.DateTime > > > and > > > > > roundTimeStamp will need to be > > > > > > com.activitystream.drill.udfs.ASUserDefinedFunctions.roundTimeStamp. > > > > > > > > > > > > > > > > > > > > On Sun, Jul 19, 2015 at 7:02 PM, Stefán Baxter < > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > The project can be found here: > > > > > > https://github.com/acmeguy/asdrill > > > > > > > > > > > > Thank you, > > > > > > -Stefán > > > > > > > > > > > > On Sun, Jul 19, 2015 at 11:57 PM, Stefán Baxter < > > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I'm more than happy to share the little that is there (I will > > > publish > > > > > it > > > > > > > on github and send link tomorrow). > > > > > > > > > > > > > > I ended up copying my UDF (singl-file.java) into the > > > > > > simple-drill-function > > > > > > > project where it got picked up. > > > > > > > > > > > > > > Then I discovered a whole new set of dependencies/limitations > > > > > > > > > > > > > > - The UDF are recompiled - any imports are invalid or at > least > > > > > > > overwritten > > > > > > > - import org.joda.time.Period; (means that Period class is > > not > > > > > > > resolved on runtime) > > > > > > > - Error: SYSTEM ERROR: CompileException: Line 71, Column 26: > > > > Cannot > > > > > > > determine simple type name "Period" > > > > > > > > > > > > > > - Calling any "outside" functions, like I was calling a > static > > > > > > > function of the new class (same file), leads to an errors as > > > they > > > > > are > > > > > > not > > > > > > > not resolved > > > > > > > - Error: SYSTEM ERROR: CompileException: Line 70, Column > 35: > > A > > > > > method > > > > > > > named "roundTimeStamp" is not declared in any enclosing > class > > > nor > > > > > any > > > > > > > supertype, nor through a static import > > > > > > > > > > > > > > Perhaps this was mentioned in the documentation but this is, at > > the > > > > > very > > > > > > > least, not straight forward and super-inviting. > > > > > > > > > > > > > > Thank you for your assistance, we will keep trying :) > > > > > > > > > > > > > > Regards, > > > > > > > -Stefan > > > > > > > > > > > > > > > > > > > > > On Sun, Jul 19, 2015 at 11:34 PM, Tugdual Grall < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > >> Hi Stefan, > > > > > > >> > > > > > > >> Do you think you can share your complete project ? > > > > > > >> > > > > > > >> This will help to debug it for you. > > > > > > >> > > > > > > >> T > > > > > > >> > > > > > > >> On Sunday, July 19, 2015, Stefán Baxter < > > > [email protected]> > > > > > > >> wrote: > > > > > > >> > > > > > > >> > Hi Ted, > > > > > > >> > > > > > > > >> > I fetched this, built it and deployed it without problems. > > > > > > >> > I can not see any real difference other than this deploys > two > > > .jar > > > > > (I > > > > > > >> tried > > > > > > >> > that as well earlier). > > > > > > >> > > > > > > > >> > I'm still trying to figure out why Drill is not picking up > my > > > UDFs > > > > > > >> > > > > > > > >> > Regards, > > > > > > >> > -Stefán > > > > > > >> > > > > > > > >> > On Sun, Jul 19, 2015 at 10:45 PM, Ted Dunning < > > > > > [email protected] > > > > > > >> > <javascript:;>> wrote: > > > > > > >> > > > > > > > >> > > Stefan, > > > > > > >> > > > > > > > > >> > > Have you seen this github project: > > > > > > >> > > > > > > > > >> > > https://github.com/mapr-demos/simple-drill-functions > > > > > > >> > > > > > > > > >> > > ? > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > On Sun, Jul 19, 2015 at 2:14 PM, Stefán Baxter < > > > > > > >> > [email protected] <javascript:;>> > > > > > > >> > > wrote: > > > > > > >> > > > > > > > > >> > > > Hi Jim, > > > > > > >> > > > > > > > > > >> > > > I'm still not able to make this work. Do you have a > sample > > > > .jar > > > > > > file > > > > > > >> > > with a > > > > > > >> > > > small example that you are running? > > > > > > >> > > > > > > > > > >> > > > Regards, > > > > > > >> > > > -Stefan > > > > > > >> > > > > > > > > > >> > > > On Sun, Jul 19, 2015 at 6:46 PM, Ted Dunning < > > > > > > [email protected] > > > > > > >> > <javascript:;>> > > > > > > >> > > > wrote: > > > > > > >> > > > > > > > > > >> > > > > > > > > > > >> > > > > Sounds like a fine example, not because of > > sophistication > > > > but > > > > > > >> because > > > > > > >> > > it > > > > > > >> > > > > deals with dates. > > > > > > >> > > > > > > > > > > >> > > > > Check the drill logs. It is likely that drill is > grumpy > > > > about > > > > > > >> > > something > > > > > > >> > > > > in your udf or packaging. > > > > > > >> > > > > > > > > > > >> > > > > Also, feel free to snitch the pom from the simple > > examples > > > > in > > > > > > >> order > > > > > > >> > to > > > > > > >> > > > get > > > > > > >> > > > > the pieces assembled and packaged correctly. > > > > > > >> > > > > > > > > > > >> > > > > Sent from my iPhone > > > > > > >> > > > > > > > > > > >> > > > > > On Jul 19, 2015, at 11:25, Stefán Baxter < > > > > > > >> > [email protected] <javascript:;>> > > > > > > >> > > > > wrote: > > > > > > >> > > > > > > > > > > > >> > > > > > Hi Jim, > > > > > > >> > > > > > > > > > > > >> > > > > > My UDF is currently so simple that I'm not sure you > > need > > > > it > > > > > > (or > > > > > > >> > want > > > > > > >> > > > it). > > > > > > >> > > > > > > > > > > > >> > > > > > It basically just rounds a timestamp value with ISO > > 8601 > > > > > > periods > > > > > > >> > > > > > asRoundTimestamp(timestampvalue,'PT10M'). > > > > > > >> > > > > > > > > > > > >> > > > > > I would be more than happy to contribute to your > > project > > > > > > rather > > > > > > >> > than > > > > > > >> > > > > > building our own :). > > > > > > >> > > > > > > > > > > > >> > > > > > Is the repo public? > > > > > > >> > > > > > > > > > > > >> > > > > > Regards, > > > > > > >> > > > > > -Stefan > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > >> On Sun, Jul 19, 2015 at 6:18 PM, Jim Bates < > > > > > > >> [email protected] > > > > > > >> > <javascript:;>> > > > > > > >> > > > wrote: > > > > > > >> > > > > >> > > > > > > >> > > > > >> Maven will typically create a jar for class and a > jar > > > for > > > > > > >> source > > > > > > >> > > when > > > > > > >> > > > > told > > > > > > >> > > > > >> to do so. I just include the source files in the > same > > > jar > > > > > as > > > > > > >> the > > > > > > >> > > class > > > > > > >> > > > > >> files. There is a github example drill udf project > we > > > are > > > > > > >> working > > > > > > >> > on > > > > > > >> > > > to > > > > > > >> > > > > >> include several examples to simplify the learning > > > curve. > > > > If > > > > > > >> your > > > > > > >> > > > > >> interested... I'd love to have you add your udf. > > > > > > >> > > > > >> On Jul 19, 2015 12:59 PM, "Stefán Baxter" < > > > > > > >> > > [email protected] <javascript:;>> > > > > > > >> > > > > >> wrote: > > > > > > >> > > > > >> > > > > > > >> > > > > >>> Hi again, > > > > > > >> > > > > >>> > > > > > > >> > > > > >>> Going over the documentation once more I came > across > > > > this: > > > > > > >> > > > > >>> > > > > > > >> > > > > >>> - Add the sources and classes JAR files to > Drill’s > > > > > > >> classpath. > > > > > > >> > > > > >>> > > > > > > >> > > > > >>> I'm only including a standard .jar (with classes > no > > > > > sources) > > > > > > >> and > > > > > > >> > I > > > > > > >> > > > > wonder > > > > > > >> > > > > >>> why this dependency is listed. > > > > > > >> > > > > >>> > > > > > > >> > > > > >>> Anyways, that it the next step. > > > > > > >> > > > > >>> > > > > > > >> > > > > >>> A sample .jar that works with the drill-config > > > > counterpart > > > > > > >> would > > > > > > >> > > be a > > > > > > >> > > > > >>> welcome addition to the documentation. > > > > > > >> > > > > >>> > > > > > > >> > > > > >>> Regards, > > > > > > >> > > > > >>> -Stefan > > > > > > >> > > > > >>> > > > > > > >> > > > > >>> > > > > > > >> > > > > >>> On Sun, Jul 19, 2015 at 5:53 PM, Stefán Baxter < > > > > > > >> > > > > >> [email protected] <javascript:;>> > > > > > > >> > > > > >>> wrote: > > > > > > >> > > > > >>> > > > > > > >> > > > > >>>> Hi Jim, > > > > > > >> > > > > >>>> > > > > > > >> > > > > >>>> Now I have added the file to the jar (both root > and > > > > > > resources > > > > > > >> > > > folder) > > > > > > >> > > > > >> but > > > > > > >> > > > > >>>> that does not seem to change anything. > > > > > > >> > > > > >>>> > > > > > > >> > > > > >>>> Any additional ideas? > > > > > > >> > > > > >>>> > > > > > > >> > > > > >>>> Regards, > > > > > > >> > > > > >>>> -Stefan > > > > > > >> > > > > >>>> > > > > > > >> > > > > >>>> On Sun, Jul 19, 2015 at 5:40 PM, Stefán Baxter < > > > > > > >> > > > > >>> [email protected] <javascript:;>> > > > > > > >> > > > > >>>> wrote: > > > > > > >> > > > > >>>> > > > > > > >> > > > > >>>>> NO! > > > > > > >> > > > > >>>>> > > > > > > >> > > > > >>>>> Thank you , will do that right now :) > > > > > > >> > > > > >>>>> > > > > > > >> > > > > >>>>> On Sun, Jul 19, 2015 at 5:38 PM, Jim Bates < > > > > > > >> > [email protected] <javascript:;>> > > > > > > >> > > > > >> wrote: > > > > > > >> > > > > >>>>> > > > > > > >> > > > > >>>>>> Did you include a file drill-module.conf in > your > > > jar > > > > > > along > > > > > > >> > with > > > > > > >> > > > > >> source > > > > > > >> > > > > >>>>>> files? > > > > > > >> > > > > >>>>>> On Jul 19, 2015 12:20 PM, "Stefán Baxter" < > > > > > > >> > > > > [email protected] <javascript:;> > > > > > > >> > > > > > > >> > > > > >>> > > > > > > >> > > > > >>>>>> wrote: > > > > > > >> > > > > >>>>>> > > > > > > >> > > > > >>>>>>> Hi, > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> I'm trying to deploy a UDF that I have written > > > > > according > > > > > > >> to > > > > > > >> > the > > > > > > >> > > > > >>>>>>> documentation. > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> I have also: > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> 1. Copied the jar file > > > > > > >> > > > > >>>>>>> to jars/3rdparty > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> 2. Changed the config > > "conf/drill-override.conf" > > > > to > > > > > > >> > include: > > > > > > >> > > > > >>>>>>> drill.logical.function.package += > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > > ["org.apache.drill.exec.expr.fn.impl","com.activitystream.drill.udfs"] > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> 3. Rerun the Drillbit > > > > > > >> > > > > >>>>>>> bin/drillbit.sh start > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> The log shows no sign of the the UDF being > > loaded > > > > but > > > > > > if I > > > > > > >> > use > > > > > > >> > > it > > > > > > >> > > > > >> in > > > > > > >> > > > > >>> a > > > > > > >> > > > > >>>>>> SQL > > > > > > >> > > > > >>>>>>> statement then this error is thrown: > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> Error: PARSE ERROR: From line 1, column 184 to > > > line > > > > 1, > > > > > > >> column > > > > > > >> > > > 219: > > > > > > >> > > > > >> No > > > > > > >> > > > > >>>>>> match > > > > > > >> > > > > >>>>>>> found for function signature > > > asRoundTimeStamp(<ANY>, > > > > > > >> > > <CHARACTER>) > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> The relevant part of the UDF source: > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> package com.activitystream.drill.udfs; > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> public class ASUserDefinedFunctions { > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> @FunctionTemplate(name = > "asRoundTimeStamp", > > > > scope > > > > > = > > > > > > >> > > > > >>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls = > > > > > > >> > > > > >>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL) > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> public static class RoundTimeStamp > implements > > > > > > >> > > DrillSimpleFunc > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> Can anyone please assist? > > > > > > >> > > > > >>>>>>> > > > > > > >> > > > > >>>>>>> Regards, > > > > > > >> > > > > >>>>>>> -Stefan > > > > > > >> > > > > >> > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
