Hi, all. I’m just a fly on the wall here, as usual. This was an interesting thread to follow. Two observations:
1. It seems to me like Drill is at a point where, if you thread the needle perfectly, things generally work as advertised. That’s certainly an advance over the old, old days, where stuff that should have worked sometimes just didn’t. 2. Threading that needle can be super-hard, even for an experienced Java developer. 3. If I were to compare all of this to something from my world — .NET CLR-based UDFs in SQL Server (2005 and later) — the fit and finish on the Drill side isn’t anywhere close. That’s not a fair comparison, of course. But it does seem to point out a goal, especially because it seems the harder work is done. Another thing to consider is that SQL Server CLR features are often disabled by DBAs, and Drill won’t have that problem. So the Drill project could take a familiar feature from the RDBMS world, and make it much more useful. 4. Overall, I’m encouraged to see all this and think it’s great that tenacious people like Stefan are willing to slog through until things work. 5. I’m seeing greater traction for Drill across the industry and expect it will grow. Keep up the good work! On 7/20/15, 10:23 AM, "Jim Bates" <[email protected]> wrote: >That's an apt description. The Holders that have a fixed size value objects >are simple to get at but the ones that have variable length objects are >pulled via the buffer. > >You can also use the StringFunctionHelpers. > >org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(in >.start, in.end, in.buffer) > >Where 'in' is > >@Param NullableVarCharHolder in > >or > >@Param VarCharHolder in > >On Mon, Jul 20, 2015 at 9:07 AM, Stefán Baxter <[email protected]> >wrote: > >> Hi, >> >> After going through the log this is clear what is happening (once the Drill >> picked up the UDF a bit earlier this morning). >> >> I'm calling the VarCharHolder.toString() to get the text value for the >> parameter and that is throwing this exception. >> >> Two observations: >> >> 1. Calling a deprecated function, even though that its not optimal, >> usually does not cause such drastic results. >> 2. As far as I can see there is no easy way to get the string value of a >> property without carving out a piece of the buffer. >> >> What am I missing here? >> >> Regards, >> -Stefan >> >> >> On Mon, Jul 20, 2015 at 12:58 PM, Jacques Nadeau <[email protected]> >> wrote: >> >> > Can you enable verbose errors at the session level? It may reveal more >> > about what is failing. >> > On Jul 20, 2015 5:32 AM, "Stefán Baxter" <[email protected]> >> > wrote: >> > >> > > Hi Jim, >> > > >> > > I have made those changes and I'm wondering if you can runs this be >> using >> > > the two .jar files that the "mvn package" places in the target >> directory? >> > > >> > > I have tried to have Drill pick those up but the error now is: >> > > >> > > Error: SYSTEM ERROR: UnsupportedOperationException >> > > Fragment 0:0 >> > > [Error Id: da589dd4-4cfd-4659-8b93-219074ab8c72 on localhost:31010] >> > > (state=,code=0) >> > > >> > > It seems to indicate that it's picking up the functions but that they >> can >> > > not be run. >> > > >> > > Regards, >> > > - Stefán >> > > >> > > On Mon, Jul 20, 2015 at 1:25 AM, Jim Bates <[email protected]> >> wrote: >> > > >> > > > I pulled out your udf class and threw it into my package. It worked >> for >> > > me >> > > > with a few modifications. >> > > > >> > > > You can not have imported classes or method references in your eval >> or >> > > > setup methods as the code will get pulled out and executed somewhere >> > else >> > > > and it won't be able to find it. With that in mind, Period will need >> > to >> > > be >> > > > org.joda.time.Period, DateTime will need to be org.joda.time.DateTime >> > and >> > > > roundTimeStamp will need to be >> > > > com.activitystream.drill.udfs.ASUserDefinedFunctions.roundTimeStamp. >> > > > >> > > > >> > > > >> > > > On Sun, Jul 19, 2015 at 7:02 PM, Stefán Baxter < >> > > [email protected]> >> > > > wrote: >> > > > >> > > > > Hi, >> > > > > >> > > > > The project can be found here: >> > > > > https://github.com/acmeguy/asdrill >> > > > > >> > > > > Thank you, >> > > > > -Stefán >> > > > > >> > > > > On Sun, Jul 19, 2015 at 11:57 PM, Stefán Baxter < >> > > > [email protected] >> > > > > > >> > > > > wrote: >> > > > > >> > > > > > Hi, >> > > > > > >> > > > > > I'm more than happy to share the little that is there (I will >> > publish >> > > > it >> > > > > > on github and send link tomorrow). >> > > > > > >> > > > > > I ended up copying my UDF (singl-file.java) into the >> > > > > simple-drill-function >> > > > > > project where it got picked up. >> > > > > > >> > > > > > Then I discovered a whole new set of dependencies/limitations >> > > > > > >> > > > > > - The UDF are recompiled - any imports are invalid or at least >> > > > > > overwritten >> > > > > > - import org.joda.time.Period; (means that Period class is >> not >> > > > > > resolved on runtime) >> > > > > > - Error: SYSTEM ERROR: CompileException: Line 71, Column 26: >> > > Cannot >> > > > > > determine simple type name "Period" >> > > > > > >> > > > > > - Calling any "outside" functions, like I was calling a static >> > > > > > function of the new class (same file), leads to an errors as >> > they >> > > > are >> > > > > not >> > > > > > not resolved >> > > > > > - Error: SYSTEM ERROR: CompileException: Line 70, Column 35: >> A >> > > > method >> > > > > > named "roundTimeStamp" is not declared in any enclosing class >> > nor >> > > > any >> > > > > > supertype, nor through a static import >> > > > > > >> > > > > > Perhaps this was mentioned in the documentation but this is, at >> the >> > > > very >> > > > > > least, not straight forward and super-inviting. >> > > > > > >> > > > > > Thank you for your assistance, we will keep trying :) >> > > > > > >> > > > > > Regards, >> > > > > > -Stefan >> > > > > > >> > > > > > >> > > > > > On Sun, Jul 19, 2015 at 11:34 PM, Tugdual Grall < >> [email protected] >> > > >> > > > > wrote: >> > > > > > >> > > > > >> Hi Stefan, >> > > > > >> >> > > > > >> Do you think you can share your complete project ? >> > > > > >> >> > > > > >> This will help to debug it for you. >> > > > > >> >> > > > > >> T >> > > > > >> >> > > > > >> On Sunday, July 19, 2015, Stefán Baxter < >> > [email protected]> >> > > > > >> wrote: >> > > > > >> >> > > > > >> > Hi Ted, >> > > > > >> > >> > > > > >> > I fetched this, built it and deployed it without problems. >> > > > > >> > I can not see any real difference other than this deploys two >> > .jar >> > > > (I >> > > > > >> tried >> > > > > >> > that as well earlier). >> > > > > >> > >> > > > > >> > I'm still trying to figure out why Drill is not picking up my >> > UDFs >> > > > > >> > >> > > > > >> > Regards, >> > > > > >> > -Stefán >> > > > > >> > >> > > > > >> > On Sun, Jul 19, 2015 at 10:45 PM, Ted Dunning < >> > > > [email protected] >> > > > > >> > <javascript:;>> wrote: >> > > > > >> > >> > > > > >> > > Stefan, >> > > > > >> > > >> > > > > >> > > Have you seen this github project: >> > > > > >> > > >> > > > > >> > > https://github.com/mapr-demos/simple-drill-functions >> > > > > >> > > >> > > > > >> > > ? >> > > > > >> > > >> > > > > >> > > >> > > > > >> > > On Sun, Jul 19, 2015 at 2:14 PM, Stefán Baxter < >> > > > > >> > [email protected] <javascript:;>> >> > > > > >> > > wrote: >> > > > > >> > > >> > > > > >> > > > Hi Jim, >> > > > > >> > > > >> > > > > >> > > > I'm still not able to make this work. Do you have a sample >> > > .jar >> > > > > file >> > > > > >> > > with a >> > > > > >> > > > small example that you are running? >> > > > > >> > > > >> > > > > >> > > > Regards, >> > > > > >> > > > -Stefan >> > > > > >> > > > >> > > > > >> > > > On Sun, Jul 19, 2015 at 6:46 PM, Ted Dunning < >> > > > > [email protected] >> > > > > >> > <javascript:;>> >> > > > > >> > > > wrote: >> > > > > >> > > > >> > > > > >> > > > > >> > > > > >> > > > > Sounds like a fine example, not because of >> sophistication >> > > but >> > > > > >> because >> > > > > >> > > it >> > > > > >> > > > > deals with dates. >> > > > > >> > > > > >> > > > > >> > > > > Check the drill logs. It is likely that drill is grumpy >> > > about >> > > > > >> > > something >> > > > > >> > > > > in your udf or packaging. >> > > > > >> > > > > >> > > > > >> > > > > Also, feel free to snitch the pom from the simple >> examples >> > > in >> > > > > >> order >> > > > > >> > to >> > > > > >> > > > get >> > > > > >> > > > > the pieces assembled and packaged correctly. >> > > > > >> > > > > >> > > > > >> > > > > Sent from my iPhone >> > > > > >> > > > > >> > > > > >> > > > > > On Jul 19, 2015, at 11:25, Stefán Baxter < >> > > > > >> > [email protected] <javascript:;>> >> > > > > >> > > > > wrote: >> > > > > >> > > > > > >> > > > > >> > > > > > Hi Jim, >> > > > > >> > > > > > >> > > > > >> > > > > > My UDF is currently so simple that I'm not sure you >> need >> > > it >> > > > > (or >> > > > > >> > want >> > > > > >> > > > it). >> > > > > >> > > > > > >> > > > > >> > > > > > It basically just rounds a timestamp value with ISO >> 8601 >> > > > > periods >> > > > > >> > > > > > asRoundTimestamp(timestampvalue,'PT10M'). >> > > > > >> > > > > > >> > > > > >> > > > > > I would be more than happy to contribute to your >> project >> > > > > rather >> > > > > >> > than >> > > > > >> > > > > > building our own :). >> > > > > >> > > > > > >> > > > > >> > > > > > Is the repo public? >> > > > > >> > > > > > >> > > > > >> > > > > > Regards, >> > > > > >> > > > > > -Stefan >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> On Sun, Jul 19, 2015 at 6:18 PM, Jim Bates < >> > > > > >> [email protected] >> > > > > >> > <javascript:;>> >> > > > > >> > > > wrote: >> > > > > >> > > > > >> >> > > > > >> > > > > >> Maven will typically create a jar for class and a jar >> > for >> > > > > >> source >> > > > > >> > > when >> > > > > >> > > > > told >> > > > > >> > > > > >> to do so. I just include the source files in the same >> > jar >> > > > as >> > > > > >> the >> > > > > >> > > class >> > > > > >> > > > > >> files. There is a github example drill udf project we >> > are >> > > > > >> working >> > > > > >> > on >> > > > > >> > > > to >> > > > > >> > > > > >> include several examples to simplify the learning >> > curve. >> > > If >> > > > > >> your >> > > > > >> > > > > >> interested... I'd love to have you add your udf. >> > > > > >> > > > > >> On Jul 19, 2015 12:59 PM, "Stefán Baxter" < >> > > > > >> > > [email protected] <javascript:;>> >> > > > > >> > > > > >> wrote: >> > > > > >> > > > > >> >> > > > > >> > > > > >>> Hi again, >> > > > > >> > > > > >>> >> > > > > >> > > > > >>> Going over the documentation once more I came across >> > > this: >> > > > > >> > > > > >>> >> > > > > >> > > > > >>> - Add the sources and classes JAR files to Drill’s >> > > > > >> classpath. >> > > > > >> > > > > >>> >> > > > > >> > > > > >>> I'm only including a standard .jar (with classes no >> > > > sources) >> > > > > >> and >> > > > > >> > I >> > > > > >> > > > > wonder >> > > > > >> > > > > >>> why this dependency is listed. >> > > > > >> > > > > >>> >> > > > > >> > > > > >>> Anyways, that it the next step. >> > > > > >> > > > > >>> >> > > > > >> > > > > >>> A sample .jar that works with the drill-config >> > > counterpart >> > > > > >> would >> > > > > >> > > be a >> > > > > >> > > > > >>> welcome addition to the documentation. >> > > > > >> > > > > >>> >> > > > > >> > > > > >>> Regards, >> > > > > >> > > > > >>> -Stefan >> > > > > >> > > > > >>> >> > > > > >> > > > > >>> >> > > > > >> > > > > >>> On Sun, Jul 19, 2015 at 5:53 PM, Stefán Baxter < >> > > > > >> > > > > >> [email protected] <javascript:;>> >> > > > > >> > > > > >>> wrote: >> > > > > >> > > > > >>> >> > > > > >> > > > > >>>> Hi Jim, >> > > > > >> > > > > >>>> >> > > > > >> > > > > >>>> Now I have added the file to the jar (both root and >> > > > > resources >> > > > > >> > > > folder) >> > > > > >> > > > > >> but >> > > > > >> > > > > >>>> that does not seem to change anything. >> > > > > >> > > > > >>>> >> > > > > >> > > > > >>>> Any additional ideas? >> > > > > >> > > > > >>>> >> > > > > >> > > > > >>>> Regards, >> > > > > >> > > > > >>>> -Stefan >> > > > > >> > > > > >>>> >> > > > > >> > > > > >>>> On Sun, Jul 19, 2015 at 5:40 PM, Stefán Baxter < >> > > > > >> > > > > >>> [email protected] <javascript:;>> >> > > > > >> > > > > >>>> wrote: >> > > > > >> > > > > >>>> >> > > > > >> > > > > >>>>> NO! >> > > > > >> > > > > >>>>> >> > > > > >> > > > > >>>>> Thank you , will do that right now :) >> > > > > >> > > > > >>>>> >> > > > > >> > > > > >>>>> On Sun, Jul 19, 2015 at 5:38 PM, Jim Bates < >> > > > > >> > [email protected] <javascript:;>> >> > > > > >> > > > > >> wrote: >> > > > > >> > > > > >>>>> >> > > > > >> > > > > >>>>>> Did you include a file drill-module.conf in your >> > jar >> > > > > along >> > > > > >> > with >> > > > > >> > > > > >> source >> > > > > >> > > > > >>>>>> files? >> > > > > >> > > > > >>>>>> On Jul 19, 2015 12:20 PM, "Stefán Baxter" < >> > > > > >> > > > > [email protected] <javascript:;> >> > > > > >> >> > > > > >> > > > > >>> >> > > > > >> > > > > >>>>>> wrote: >> > > > > >> > > > > >>>>>> >> > > > > >> > > > > >>>>>>> Hi, >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> I'm trying to deploy a UDF that I have written >> > > > according >> > > > > >> to >> > > > > >> > the >> > > > > >> > > > > >>>>>>> documentation. >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> I have also: >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> 1. Copied the jar file >> > > > > >> > > > > >>>>>>> to jars/3rdparty >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> 2. Changed the config >> "conf/drill-override.conf" >> > > to >> > > > > >> > include: >> > > > > >> > > > > >>>>>>> drill.logical.function.package += >> > > > > >> > > > > >> >> > > > > >> > > >> > > > > >> > ["org.apache.drill.exec.expr.fn.impl","com.activitystream.drill.udfs"] >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> 3. Rerun the Drillbit >> > > > > >> > > > > >>>>>>> bin/drillbit.sh start >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> The log shows no sign of the the UDF being >> loaded >> > > but >> > > > > if I >> > > > > >> > use >> > > > > >> > > it >> > > > > >> > > > > >> in >> > > > > >> > > > > >>> a >> > > > > >> > > > > >>>>>> SQL >> > > > > >> > > > > >>>>>>> statement then this error is thrown: >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> Error: PARSE ERROR: From line 1, column 184 to >> > line >> > > 1, >> > > > > >> column >> > > > > >> > > > 219: >> > > > > >> > > > > >> No >> > > > > >> > > > > >>>>>> match >> > > > > >> > > > > >>>>>>> found for function signature >> > asRoundTimeStamp(<ANY>, >> > > > > >> > > <CHARACTER>) >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> The relevant part of the UDF source: >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> package com.activitystream.drill.udfs; >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> public class ASUserDefinedFunctions { >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> @FunctionTemplate(name = "asRoundTimeStamp", >> > > scope >> > > > = >> > > > > >> > > > > >>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls = >> > > > > >> > > > > >>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL) >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> public static class RoundTimeStamp implements >> > > > > >> > > DrillSimpleFunc >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> Can anyone please assist? >> > > > > >> > > > > >>>>>>> >> > > > > >> > > > > >>>>>>> Regards, >> > > > > >> > > > > >>>>>>> -Stefan >> > > > > >> > > > > >> >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >>
