On Tue, Jun 14, 2016 at 10:23 AM, Sachith Withana <[email protected]> wrote:
> What do you mean by workaround? > > On Tue, Jun 14, 2016 at 9:16 AM, Dulitha Wijewantha <[email protected]> > wrote: > >> >> >> On Sat, Jun 11, 2016 at 12:33 AM, Sachith Withana <[email protected]> >> wrote: >> >>> If you need to skip the rows that would be throwing the exception, as >>> you said, you'd have to do that in a where clause. >>> If there's a scenario where you need to transform the values and use in >>> a SELECT statement, use a subquery and filter out the unwanted rows. >>> >> >> Thanks. This can be used as a workaround. >> >> >> > By having the query inside an inner query and filtering it out using a WHERE clause (this is the workaround). IMO - a solution for this would be to allow UDFs to propagate exceptions that will change the behavior (ignore the current row). But in the future do you plan to incorporate this as a feature where I can >> throw a particular exception to from the UDF that will skip that record or >> stop the whole script? (Is this part of the Spark-SQL code too) >> >> >> >>> >>> ex: >>> >>> SELECT date FROM ( SELECT dateConvertor(timestamp, "MM/dd/yyyy HH:mm", >>> "HOUR") as date FROM eventStatData) Where date != ""; >>> >>> On Fri, Jun 10, 2016 at 2:25 PM, Dulitha Wijewantha <[email protected]> >>> wrote: >>> >>>> The default value will work for scenarios where we are using the >>>> function on a where clause. There is also another scenario where you use it >>>> to transform the value in the select. In the below query - if we put a >>>> default value it will be returned by the result set rather than ignoring >>>> it. >>>> >>>> select sessionKey as sessionKey, dateFormatter(timestamp, "MM/dd/yyyy >>>> HH:mm", "yyyy-MM-dd:HH") as dateDisplay, appKey as appKey, clientModel as >>>> role, dateConvertor(timestamp, "MM/dd/yyyy HH:mm", "HOUR") as date from >>>> eventStatData >>>> >>>> On Fri, Jun 10, 2016 at 12:10 AM, Gimantha Bandara <[email protected]> >>>> wrote: >>>> >>>>> You will have to catch the exception inside the method itself and log >>>>> a warning/error meesage and return a default value without throwing the >>>>> exception out from the method. >>>>> >>>>> >>>>> On Friday, June 10, 2016, Dulitha Wijewantha <[email protected]> wrote: >>>>> >>>>>> Below is an example usage - >>>>>> >>>>>> insert overwrite table TABLE_NAME select date, count(distinct >>>>>> sessionKey) as activity, appKey, role, first(y.dateDisplay) as >>>>>> dateDisplay >>>>>> from (select sessionKey as sessionKey, dateFormatter(timestamp, >>>>>> "MM/dd/yyyy HH:mm", "yyyy-MM-dd:HH") as dateDisplay, appKey as appKey, >>>>>> clientModel as role, dateConvertor(timestamp, "MM/dd/yyyy HH:mm", "HOUR") >>>>>> as date from eventStatData) as y group by y.date, y.appKey, y.role; >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 9, 2016 at 4:13 PM, Sachith Withana <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Dulitha, >>>>>>> >>>>>>> Can you provide the query that you are using? >>>>>>> >>>>>>> That would help us give you a better solution I guess. >>>>>>> >>>>>>> Regards, >>>>>>> Sachith >>>>>>> >>>>>>> On Thu, Jun 9, 2016 at 2:34 PM, Dulitha Wijewantha <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> Hi guys, >>>>>>>> I have written a customer Spark UDF function that allows me to >>>>>>>> convert a date string to a given input format. If there is a >>>>>>>> ParseException >>>>>>>> due to input being invalid - the whole spark script will stop >>>>>>>> execution. Is >>>>>>>> there away to skip that particular iteration and move on to the next >>>>>>>> in the >>>>>>>> record? (this function is used in where clauses). Or is there a better >>>>>>>> recommendation for this? Like validation in the EventReceiver? >>>>>>>> >>>>>>>> public String dateFormatter(String input, String fromFormat, String >>>>>>>> toFormat) throws ParseException{ >>>>>>>> SimpleDateFormat sdf = new SimpleDateFormat(fromFormat); >>>>>>>> Date date = sdf.parse(input); >>>>>>>> SimpleDateFormat newFormatter = new SimpleDateFormat(toFormat); >>>>>>>> return newFormatter.format(date); >>>>>>>> } >>>>>>>> >>>>>>>> Cheers~ >>>>>>>> >>>>>>>> -- >>>>>>>> Dulitha Wijewantha (Chan) >>>>>>>> Software Engineer - Mobile Development >>>>>>>> WSO2 Inc >>>>>>>> Lean.Enterprise.Middleware >>>>>>>> * ~Email [email protected]* >>>>>>>> * ~Mobile +94712112165 <%2B94712112165>* >>>>>>>> * ~Website dulitha.me <http://dulitha.me>* >>>>>>>> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >>>>>>>> *~Github @dulichan <https://github.com/dulichan>* >>>>>>>> *~SO @chan <http://stackoverflow.com/users/813471/chan>* >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Sachith Withana >>>>>>> Software Engineer; WSO2 Inc.; http://wso2.com >>>>>>> E-mail: sachith AT wso2.com >>>>>>> M: +94715518127 >>>>>>> Linked-In: <http://goog_416592669> >>>>>>> https://lk.linkedin.com/in/sachithwithana >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Dulitha Wijewantha (Chan) >>>>>> Software Engineer - Mobile Development >>>>>> WSO2 Inc >>>>>> Lean.Enterprise.Middleware >>>>>> * ~Email [email protected]* >>>>>> * ~Mobile +94712112165 <%2B94712112165>* >>>>>> * ~Website dulitha.me <http://dulitha.me>* >>>>>> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >>>>>> *~Github @dulichan <https://github.com/dulichan>* >>>>>> *~SO @chan <http://stackoverflow.com/users/813471/chan>* >>>>>> >>>>> >>>>> >>>>> -- >>>>> Gimantha Bandara >>>>> Software Engineer >>>>> WSO2. Inc : http://wso2.com >>>>> Mobile : +94714961919 >>>>> >>>>> >>>> >>>> >>>> -- >>>> Dulitha Wijewantha (Chan) >>>> Software Engineer - Mobile Development >>>> WSO2 Inc >>>> Lean.Enterprise.Middleware >>>> * ~Email [email protected] <[email protected]>* >>>> * ~Mobile +94712112165 <%2B94712112165>* >>>> * ~Website dulitha.me <http://dulitha.me>* >>>> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >>>> *~Github @dulichan <https://github.com/dulichan>* >>>> *~SO @chan <http://stackoverflow.com/users/813471/chan>* >>>> >>> >>> >>> >>> -- >>> Sachith Withana >>> Software Engineer; WSO2 Inc.; http://wso2.com >>> E-mail: sachith AT wso2.com >>> M: +94715518127 >>> Linked-In: <http://goog_416592669> >>> https://lk.linkedin.com/in/sachithwithana >>> >> >> >> >> -- >> Dulitha Wijewantha (Chan) >> Software Engineer - Mobile Development >> WSO2 Inc >> Lean.Enterprise.Middleware >> * ~Email [email protected] <[email protected]>* >> * ~Mobile +94712112165 <%2B94712112165>* >> * ~Website dulitha.me <http://dulitha.me>* >> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >> *~Github @dulichan <https://github.com/dulichan>* >> *~SO @chan <http://stackoverflow.com/users/813471/chan>* >> > > > > -- > Sachith Withana > Software Engineer; WSO2 Inc.; http://wso2.com > E-mail: sachith AT wso2.com > M: +94715518127 > Linked-In: <http://goog_416592669> > https://lk.linkedin.com/in/sachithwithana > -- Dulitha Wijewantha (Chan) Software Engineer - Mobile Development WSO2 Inc Lean.Enterprise.Middleware * ~Email [email protected] <[email protected]>* * ~Mobile +94712112165* * ~Website dulitha.me <http://dulitha.me>* * ~Twitter @dulitharw <https://twitter.com/dulitharw>* *~Github @dulichan <https://github.com/dulichan>* *~SO @chan <http://stackoverflow.com/users/813471/chan>*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
