> > Somehow there are issues in implementing certain wrangler functions due to > limitations in JavaRDD used in spark > e.g. - > Fill operation - when filling with values from rows above and below > Fold operation
Agree, since rows will get executed randomly with spark, inter-row operations are not very meaningful. But you can slightly modify the implementation of the "Fill" operation, such as, to fill values based on an expression/static-value/mean etc. (not depending on other rows).. Thanks, Supun On Tue, Jun 16, 2015 at 9:27 AM, Supun Sethunga <[email protected]> wrote: > Hi Danula, > > Sorry for the late reply. Have you got the details you were looking for? > > It would be great if I could get to know which wrangler operations are >> important for a user of the ML > > > Other than the ones you have mentioned in the proposal, think its better > to have "Translate" operation as well (to create a new column based on an > existing column). > > Thanks, > Supun > > > > On Thu, Jun 4, 2015 at 10:11 PM, Danula Eranjith <[email protected]> > wrote: > >> Hi all, >> >> I am currently working on generating spark transformations related to the >> operations available in the data wrangler. >> >> Data wrangler provides sufficient parameters to re-create these at >> spark.I have successfully implemented delete and split operations of >> wrangler in spark. >> >> Once this phase is completed, I can either directly generate these >> scripts at wrangler or use the javascript output and convert it to spark >> depending on the implementation. >> >> Somehow there are issues in implementing certain wrangler functions due >> to limitations in JavaRDD used in spark >> >> e.g. - >> Fill operation - when filling with values from rows above and below >> Fold operation >> >> It would be great if I could get to know which wrangler operations are >> important for a user of the ML >> >> Thanks, >> Danula >> >> On Wed, Jun 3, 2015 at 8:30 AM, Nirmal Fernando <[email protected]> wrote: >> >>> Hi Danula, >>> >>> Please send an update of your work thus far. >>> >>> On Sun, May 10, 2015 at 2:30 PM, Nirmal Fernando <[email protected]> >>> wrote: >>> >>>> Hi Danula, >>>> >>>> Welcome to GSoC 15' ! Can you do some research on directly generating >>>> spark transformations using Wrangler and come up with a summary ? >>>> >>>> On Fri, May 8, 2015 at 11:03 AM, Danula Eranjith <[email protected]> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Thank you for selecting my proposal [1] >>>>> <https://docs.google.com/document/d/18NFa23CrhXqnHrkl_AuRz3sQ3Axg7SEmiA7l66Hl9_0/edit?usp=sharing> >>>>> for GSoC 2015. I am really looking forward to work with you all and >>>>> contribute to WSO2. >>>>> >>>>> I have already completed my primary research on wrangler and would >>>>> like to meet you to get feedback on the proposed architecture. I am >>>>> planning to start working on the project before 25th of May. >>>>> >>>>> Thank you, >>>>> Danula >>>>> >>>>> [1] - >>>>> https://docs.google.com/document/d/18NFa23CrhXqnHrkl_AuRz3sQ3Axg7SEmiA7l66Hl9_0/edit?usp=sharing >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> > > > -- > *Supun Sethunga* > Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
