(Heh, sorry, guess that was *five* observations :-) )
On 7/20/15, 11:22 AM, "Andrew Brust" <[email protected]> wrote: >Hi, all. I’m just a fly on the wall here, as usual. This was an interesting >thread to follow. Two observations: > >1. It seems to me like Drill is at a point where, if you thread the needle >perfectly, things generally work as advertised. That’s certainly an advance >over the old, old days, where stuff that should have worked sometimes just >didn’t. >2. Threading that needle can be super-hard, even for an experienced Java >developer. >3. If I were to compare all of this to something from my world — .NET >CLR-based UDFs in SQL Server (2005 and later) — the fit and finish on the >Drill side isn’t anywhere close. That’s not a fair comparison, of course. >But it does seem to point out a goal, especially because it seems the harder >work is done. Another thing to consider is that SQL Server CLR features are >often disabled by DBAs, and Drill won’t have that problem. So the Drill >project could take a familiar feature from the RDBMS world, and make it much >more useful. >4. Overall, I’m encouraged to see all this and think it’s great that tenacious >people like Stefan are willing to slog through until things work. >5. I’m seeing greater traction for Drill across the industry and expect it >will grow. Keep up the good work! > > > >On 7/20/15, 10:23 AM, "Jim Bates" <[email protected]> wrote: > >>That's an apt description. The Holders that have a fixed size value objects >>are simple to get at but the ones that have variable length objects are >>pulled via the buffer. >> >>You can also use the StringFunctionHelpers. >> >>org.apache.drill.exec.expr.fn.impl.StringFunctionHelpers.toStringFromUTF8(in >>.start, in.end, in.buffer) >> >>Where 'in' is >> >>@Param NullableVarCharHolder in >> >>or >> >>@Param VarCharHolder in >> >>On Mon, Jul 20, 2015 at 9:07 AM, Stefán Baxter <[email protected]> >>wrote: >> >>> Hi, >>> >>> After going through the log this is clear what is happening (once the Drill >>> picked up the UDF a bit earlier this morning). >>> >>> I'm calling the VarCharHolder.toString() to get the text value for the >>> parameter and that is throwing this exception. >>> >>> Two observations: >>> >>> 1. Calling a deprecated function, even though that its not optimal, >>> usually does not cause such drastic results. >>> 2. As far as I can see there is no easy way to get the string value of a >>> property without carving out a piece of the buffer. >>> >>> What am I missing here? >>> >>> Regards, >>> -Stefan >>> >>> >>> On Mon, Jul 20, 2015 at 12:58 PM, Jacques Nadeau <[email protected]> >>> wrote: >>> >>> > Can you enable verbose errors at the session level? It may reveal more >>> > about what is failing. >>> > On Jul 20, 2015 5:32 AM, "Stefán Baxter" <[email protected]> >>> > wrote: >>> > >>> > > Hi Jim, >>> > > >>> > > I have made those changes and I'm wondering if you can runs this be >>> using >>> > > the two .jar files that the "mvn package" places in the target >>> directory? >>> > > >>> > > I have tried to have Drill pick those up but the error now is: >>> > > >>> > > Error: SYSTEM ERROR: UnsupportedOperationException >>> > > Fragment 0:0 >>> > > [Error Id: da589dd4-4cfd-4659-8b93-219074ab8c72 on localhost:31010] >>> > > (state=,code=0) >>> > > >>> > > It seems to indicate that it's picking up the functions but that they >>> can >>> > > not be run. >>> > > >>> > > Regards, >>> > > - Stefán >>> > > >>> > > On Mon, Jul 20, 2015 at 1:25 AM, Jim Bates <[email protected]> >>> wrote: >>> > > >>> > > > I pulled out your udf class and threw it into my package. It worked >>> for >>> > > me >>> > > > with a few modifications. >>> > > > >>> > > > You can not have imported classes or method references in your eval >>> or >>> > > > setup methods as the code will get pulled out and executed somewhere >>> > else >>> > > > and it won't be able to find it. With that in mind, Period will need >>> > to >>> > > be >>> > > > org.joda.time.Period, DateTime will need to be org.joda.time.DateTime >>> > and >>> > > > roundTimeStamp will need to be >>> > > > com.activitystream.drill.udfs.ASUserDefinedFunctions.roundTimeStamp. >>> > > > >>> > > > >>> > > > >>> > > > On Sun, Jul 19, 2015 at 7:02 PM, Stefán Baxter < >>> > > [email protected]> >>> > > > wrote: >>> > > > >>> > > > > Hi, >>> > > > > >>> > > > > The project can be found here: >>> > > > > https://github.com/acmeguy/asdrill >>> > > > > >>> > > > > Thank you, >>> > > > > -Stefán >>> > > > > >>> > > > > On Sun, Jul 19, 2015 at 11:57 PM, Stefán Baxter < >>> > > > [email protected] >>> > > > > > >>> > > > > wrote: >>> > > > > >>> > > > > > Hi, >>> > > > > > >>> > > > > > I'm more than happy to share the little that is there (I will >>> > publish >>> > > > it >>> > > > > > on github and send link tomorrow). >>> > > > > > >>> > > > > > I ended up copying my UDF (singl-file.java) into the >>> > > > > simple-drill-function >>> > > > > > project where it got picked up. >>> > > > > > >>> > > > > > Then I discovered a whole new set of dependencies/limitations >>> > > > > > >>> > > > > > - The UDF are recompiled - any imports are invalid or at least >>> > > > > > overwritten >>> > > > > > - import org.joda.time.Period; (means that Period class is >>> not >>> > > > > > resolved on runtime) >>> > > > > > - Error: SYSTEM ERROR: CompileException: Line 71, Column 26: >>> > > Cannot >>> > > > > > determine simple type name "Period" >>> > > > > > >>> > > > > > - Calling any "outside" functions, like I was calling a static >>> > > > > > function of the new class (same file), leads to an errors as >>> > they >>> > > > are >>> > > > > not >>> > > > > > not resolved >>> > > > > > - Error: SYSTEM ERROR: CompileException: Line 70, Column 35: >>> A >>> > > > method >>> > > > > > named "roundTimeStamp" is not declared in any enclosing class >>> > nor >>> > > > any >>> > > > > > supertype, nor through a static import >>> > > > > > >>> > > > > > Perhaps this was mentioned in the documentation but this is, at >>> the >>> > > > very >>> > > > > > least, not straight forward and super-inviting. >>> > > > > > >>> > > > > > Thank you for your assistance, we will keep trying :) >>> > > > > > >>> > > > > > Regards, >>> > > > > > -Stefan >>> > > > > > >>> > > > > > >>> > > > > > On Sun, Jul 19, 2015 at 11:34 PM, Tugdual Grall < >>> [email protected] >>> > > >>> > > > > wrote: >>> > > > > > >>> > > > > >> Hi Stefan, >>> > > > > >> >>> > > > > >> Do you think you can share your complete project ? >>> > > > > >> >>> > > > > >> This will help to debug it for you. >>> > > > > >> >>> > > > > >> T >>> > > > > >> >>> > > > > >> On Sunday, July 19, 2015, Stefán Baxter < >>> > [email protected]> >>> > > > > >> wrote: >>> > > > > >> >>> > > > > >> > Hi Ted, >>> > > > > >> > >>> > > > > >> > I fetched this, built it and deployed it without problems. >>> > > > > >> > I can not see any real difference other than this deploys two >>> > .jar >>> > > > (I >>> > > > > >> tried >>> > > > > >> > that as well earlier). >>> > > > > >> > >>> > > > > >> > I'm still trying to figure out why Drill is not picking up my >>> > UDFs >>> > > > > >> > >>> > > > > >> > Regards, >>> > > > > >> > -Stefán >>> > > > > >> > >>> > > > > >> > On Sun, Jul 19, 2015 at 10:45 PM, Ted Dunning < >>> > > > [email protected] >>> > > > > >> > <javascript:;>> wrote: >>> > > > > >> > >>> > > > > >> > > Stefan, >>> > > > > >> > > >>> > > > > >> > > Have you seen this github project: >>> > > > > >> > > >>> > > > > >> > > https://github.com/mapr-demos/simple-drill-functions >>> > > > > >> > > >>> > > > > >> > > ? >>> > > > > >> > > >>> > > > > >> > > >>> > > > > >> > > On Sun, Jul 19, 2015 at 2:14 PM, Stefán Baxter < >>> > > > > >> > [email protected] <javascript:;>> >>> > > > > >> > > wrote: >>> > > > > >> > > >>> > > > > >> > > > Hi Jim, >>> > > > > >> > > > >>> > > > > >> > > > I'm still not able to make this work. Do you have a sample >>> > > .jar >>> > > > > file >>> > > > > >> > > with a >>> > > > > >> > > > small example that you are running? >>> > > > > >> > > > >>> > > > > >> > > > Regards, >>> > > > > >> > > > -Stefan >>> > > > > >> > > > >>> > > > > >> > > > On Sun, Jul 19, 2015 at 6:46 PM, Ted Dunning < >>> > > > > [email protected] >>> > > > > >> > <javascript:;>> >>> > > > > >> > > > wrote: >>> > > > > >> > > > >>> > > > > >> > > > > >>> > > > > >> > > > > Sounds like a fine example, not because of >>> sophistication >>> > > but >>> > > > > >> because >>> > > > > >> > > it >>> > > > > >> > > > > deals with dates. >>> > > > > >> > > > > >>> > > > > >> > > > > Check the drill logs. It is likely that drill is grumpy >>> > > about >>> > > > > >> > > something >>> > > > > >> > > > > in your udf or packaging. >>> > > > > >> > > > > >>> > > > > >> > > > > Also, feel free to snitch the pom from the simple >>> examples >>> > > in >>> > > > > >> order >>> > > > > >> > to >>> > > > > >> > > > get >>> > > > > >> > > > > the pieces assembled and packaged correctly. >>> > > > > >> > > > > >>> > > > > >> > > > > Sent from my iPhone >>> > > > > >> > > > > >>> > > > > >> > > > > > On Jul 19, 2015, at 11:25, Stefán Baxter < >>> > > > > >> > [email protected] <javascript:;>> >>> > > > > >> > > > > wrote: >>> > > > > >> > > > > > >>> > > > > >> > > > > > Hi Jim, >>> > > > > >> > > > > > >>> > > > > >> > > > > > My UDF is currently so simple that I'm not sure you >>> need >>> > > it >>> > > > > (or >>> > > > > >> > want >>> > > > > >> > > > it). >>> > > > > >> > > > > > >>> > > > > >> > > > > > It basically just rounds a timestamp value with ISO >>> 8601 >>> > > > > periods >>> > > > > >> > > > > > asRoundTimestamp(timestampvalue,'PT10M'). >>> > > > > >> > > > > > >>> > > > > >> > > > > > I would be more than happy to contribute to your >>> project >>> > > > > rather >>> > > > > >> > than >>> > > > > >> > > > > > building our own :). >>> > > > > >> > > > > > >>> > > > > >> > > > > > Is the repo public? >>> > > > > >> > > > > > >>> > > > > >> > > > > > Regards, >>> > > > > >> > > > > > -Stefan >>> > > > > >> > > > > > >>> > > > > >> > > > > > >>> > > > > >> > > > > > >>> > > > > >> > > > > >> On Sun, Jul 19, 2015 at 6:18 PM, Jim Bates < >>> > > > > >> [email protected] >>> > > > > >> > <javascript:;>> >>> > > > > >> > > > wrote: >>> > > > > >> > > > > >> >>> > > > > >> > > > > >> Maven will typically create a jar for class and a jar >>> > for >>> > > > > >> source >>> > > > > >> > > when >>> > > > > >> > > > > told >>> > > > > >> > > > > >> to do so. I just include the source files in the same >>> > jar >>> > > > as >>> > > > > >> the >>> > > > > >> > > class >>> > > > > >> > > > > >> files. There is a github example drill udf project we >>> > are >>> > > > > >> working >>> > > > > >> > on >>> > > > > >> > > > to >>> > > > > >> > > > > >> include several examples to simplify the learning >>> > curve. >>> > > If >>> > > > > >> your >>> > > > > >> > > > > >> interested... I'd love to have you add your udf. >>> > > > > >> > > > > >> On Jul 19, 2015 12:59 PM, "Stefán Baxter" < >>> > > > > >> > > [email protected] <javascript:;>> >>> > > > > >> > > > > >> wrote: >>> > > > > >> > > > > >> >>> > > > > >> > > > > >>> Hi again, >>> > > > > >> > > > > >>> >>> > > > > >> > > > > >>> Going over the documentation once more I came across >>> > > this: >>> > > > > >> > > > > >>> >>> > > > > >> > > > > >>> - Add the sources and classes JAR files to Drill’s >>> > > > > >> classpath. >>> > > > > >> > > > > >>> >>> > > > > >> > > > > >>> I'm only including a standard .jar (with classes no >>> > > > sources) >>> > > > > >> and >>> > > > > >> > I >>> > > > > >> > > > > wonder >>> > > > > >> > > > > >>> why this dependency is listed. >>> > > > > >> > > > > >>> >>> > > > > >> > > > > >>> Anyways, that it the next step. >>> > > > > >> > > > > >>> >>> > > > > >> > > > > >>> A sample .jar that works with the drill-config >>> > > counterpart >>> > > > > >> would >>> > > > > >> > > be a >>> > > > > >> > > > > >>> welcome addition to the documentation. >>> > > > > >> > > > > >>> >>> > > > > >> > > > > >>> Regards, >>> > > > > >> > > > > >>> -Stefan >>> > > > > >> > > > > >>> >>> > > > > >> > > > > >>> >>> > > > > >> > > > > >>> On Sun, Jul 19, 2015 at 5:53 PM, Stefán Baxter < >>> > > > > >> > > > > >> [email protected] <javascript:;>> >>> > > > > >> > > > > >>> wrote: >>> > > > > >> > > > > >>> >>> > > > > >> > > > > >>>> Hi Jim, >>> > > > > >> > > > > >>>> >>> > > > > >> > > > > >>>> Now I have added the file to the jar (both root and >>> > > > > resources >>> > > > > >> > > > folder) >>> > > > > >> > > > > >> but >>> > > > > >> > > > > >>>> that does not seem to change anything. >>> > > > > >> > > > > >>>> >>> > > > > >> > > > > >>>> Any additional ideas? >>> > > > > >> > > > > >>>> >>> > > > > >> > > > > >>>> Regards, >>> > > > > >> > > > > >>>> -Stefan >>> > > > > >> > > > > >>>> >>> > > > > >> > > > > >>>> On Sun, Jul 19, 2015 at 5:40 PM, Stefán Baxter < >>> > > > > >> > > > > >>> [email protected] <javascript:;>> >>> > > > > >> > > > > >>>> wrote: >>> > > > > >> > > > > >>>> >>> > > > > >> > > > > >>>>> NO! >>> > > > > >> > > > > >>>>> >>> > > > > >> > > > > >>>>> Thank you , will do that right now :) >>> > > > > >> > > > > >>>>> >>> > > > > >> > > > > >>>>> On Sun, Jul 19, 2015 at 5:38 PM, Jim Bates < >>> > > > > >> > [email protected] <javascript:;>> >>> > > > > >> > > > > >> wrote: >>> > > > > >> > > > > >>>>> >>> > > > > >> > > > > >>>>>> Did you include a file drill-module.conf in your >>> > jar >>> > > > > along >>> > > > > >> > with >>> > > > > >> > > > > >> source >>> > > > > >> > > > > >>>>>> files? >>> > > > > >> > > > > >>>>>> On Jul 19, 2015 12:20 PM, "Stefán Baxter" < >>> > > > > >> > > > > [email protected] <javascript:;> >>> > > > > >> >>> > > > > >> > > > > >>> >>> > > > > >> > > > > >>>>>> wrote: >>> > > > > >> > > > > >>>>>> >>> > > > > >> > > > > >>>>>>> Hi, >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> I'm trying to deploy a UDF that I have written >>> > > > according >>> > > > > >> to >>> > > > > >> > the >>> > > > > >> > > > > >>>>>>> documentation. >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> I have also: >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> 1. Copied the jar file >>> > > > > >> > > > > >>>>>>> to jars/3rdparty >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> 2. Changed the config >>> "conf/drill-override.conf" >>> > > to >>> > > > > >> > include: >>> > > > > >> > > > > >>>>>>> drill.logical.function.package += >>> > > > > >> > > > > >> >>> > > > > >> > > >>> > > > > >>> > ["org.apache.drill.exec.expr.fn.impl","com.activitystream.drill.udfs"] >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> 3. Rerun the Drillbit >>> > > > > >> > > > > >>>>>>> bin/drillbit.sh start >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> The log shows no sign of the the UDF being >>> loaded >>> > > but >>> > > > > if I >>> > > > > >> > use >>> > > > > >> > > it >>> > > > > >> > > > > >> in >>> > > > > >> > > > > >>> a >>> > > > > >> > > > > >>>>>> SQL >>> > > > > >> > > > > >>>>>>> statement then this error is thrown: >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> Error: PARSE ERROR: From line 1, column 184 to >>> > line >>> > > 1, >>> > > > > >> column >>> > > > > >> > > > 219: >>> > > > > >> > > > > >> No >>> > > > > >> > > > > >>>>>> match >>> > > > > >> > > > > >>>>>>> found for function signature >>> > asRoundTimeStamp(<ANY>, >>> > > > > >> > > <CHARACTER>) >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> The relevant part of the UDF source: >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> package com.activitystream.drill.udfs; >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> public class ASUserDefinedFunctions { >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> @FunctionTemplate(name = "asRoundTimeStamp", >>> > > scope >>> > > > = >>> > > > > >> > > > > >>>>>>> FunctionTemplate.FunctionScope.SIMPLE, nulls = >>> > > > > >> > > > > >>>>>>> FunctionTemplate.NullHandling.NULL_IF_NULL) >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> public static class RoundTimeStamp implements >>> > > > > >> > > DrillSimpleFunc >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> Can anyone please assist? >>> > > > > >> > > > > >>>>>>> >>> > > > > >> > > > > >>>>>>> Regards, >>> > > > > >> > > > > >>>>>>> -Stefan >>> > > > > >> > > > > >> >>> > > > > >> > > > > >>> > > > > >> > > > >>> > > > > >> > > >>> > > > > >> > >>> > > > > >> >>> > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>>
