Yes. A UDF could be built that constructs row objects and sends them to an external function. There are some issues since you need to somehow know what columns and types are being passed.
A more promising approach I think is dependent on Arrow succeeding. At that point, you can define a batch UDF which takes a bunch of VV's as input and passes these to a native function in a language that supports VV's (such as Python). This is a little bit like the use of shell functions in Hadoop, but is far more efficient. It also deals with the problem of Jython not really being Python. On Thu, Jun 16, 2016 at 11:19 AM, Paul Rogers <[email protected]> wrote: > Ted refers to source code transformation. Drill gains its speed from value > vectors. However, VVs are a far cry from the row-based interface that most > mere mortals are accustomed to using. Since VVs are very type specific, > code is typically generated to handle the specifics of each type. Accessing > VVs in Jython may be a bit of a challenge because of the "impedence > mismatch" between how VVs work and the row-and-column view expected by most > (non-Drill) developers. > > I wonder if we've considered providing a row-oriented "facade" that can be > used by roll-your own data sources and user-defined row transforms? Might > be a hiccup in the fast VV pipeline, but might be handy for users willing > to trade a bit of speed for convenience. With such a facade, the Jython row > transforms that John mentions could be quite simple. > > On Thu, Jun 16, 2016 at 10:36 AM, Ted Dunning <[email protected]> > wrote: > > > Since UDF's use source code transformation, using Jython would be > > difficult. > > > > > > > > On Thu, Jun 16, 2016 at 9:42 AM, Arina Yelchiyeva < > > [email protected]> wrote: > > > > > Hi Charles, > > > > > > not that I am aware of. Proposed solution doesn't invent anything new, > > just > > > adds possibility to add UDFs without drillbit restart. But > contributions > > > are welcomed. > > > > > > On Thu, Jun 16, 2016 at 4:52 PM Charles Givre <[email protected]> > wrote: > > > > > > > Arina, > > > > Has there been any discussion about making it possible via Jython or > > > > something for users to write simple UDFs in Python? > > > > My ideal would be to have this capability integrated in the web GUI > > such > > > > that a user could write their UDF (in Python) right there, submit it > > and > > > it > > > > would be deployed to Drill if it passes validation tests. > > > > —C > > > > > > > > > > > > > On Jun 16, 2016, at 09:34, Arina Yelchiyeva < > > > [email protected]> > > > > wrote: > > > > > > > > > > Hi all! > > > > > > > > > > I have created Jira to allow dynamic UDFs support in Drill ( > > > > > https://issues.apache.org/jira/browse/DRILL-4726). There is a link > > to > > > > > design document in Jira description. > > > > > Comments or suggestions are welcomed. > > > > > > > > > > Kind regards > > > > > Arina > > > > > > > > > > > > > >
