On Sat, Jun 11, 2016 at 12:33 AM, Sachith Withana <[email protected]> wrote:
> If you need to skip the rows that would be throwing the exception, as you > said, you'd have to do that in a where clause. > If there's a scenario where you need to transform the values and use in a > SELECT statement, use a subquery and filter out the unwanted rows. > Thanks. This can be used as a workaround. But in the future do you plan to incorporate this as a feature where I can throw a particular exception to from the UDF that will skip that record or stop the whole script? (Is this part of the Spark-SQL code too) > > ex: > > SELECT date FROM ( SELECT dateConvertor(timestamp, "MM/dd/yyyy HH:mm", > "HOUR") as date FROM eventStatData) Where date != ""; > > On Fri, Jun 10, 2016 at 2:25 PM, Dulitha Wijewantha <[email protected]> > wrote: > >> The default value will work for scenarios where we are using the function >> on a where clause. There is also another scenario where you use it to >> transform the value in the select. In the below query - if we put a default >> value it will be returned by the result set rather than ignoring it. >> >> select sessionKey as sessionKey, dateFormatter(timestamp, "MM/dd/yyyy >> HH:mm", "yyyy-MM-dd:HH") as dateDisplay, appKey as appKey, clientModel as >> role, dateConvertor(timestamp, "MM/dd/yyyy HH:mm", "HOUR") as date from >> eventStatData >> >> On Fri, Jun 10, 2016 at 12:10 AM, Gimantha Bandara <[email protected]> >> wrote: >> >>> You will have to catch the exception inside the method itself and log a >>> warning/error meesage and return a default value without throwing the >>> exception out from the method. >>> >>> >>> On Friday, June 10, 2016, Dulitha Wijewantha <[email protected]> wrote: >>> >>>> Below is an example usage - >>>> >>>> insert overwrite table TABLE_NAME select date, count(distinct >>>> sessionKey) as activity, appKey, role, first(y.dateDisplay) as dateDisplay >>>> from (select sessionKey as sessionKey, dateFormatter(timestamp, >>>> "MM/dd/yyyy HH:mm", "yyyy-MM-dd:HH") as dateDisplay, appKey as appKey, >>>> clientModel as role, dateConvertor(timestamp, "MM/dd/yyyy HH:mm", "HOUR") >>>> as date from eventStatData) as y group by y.date, y.appKey, y.role; >>>> >>>> >>>> >>>> On Thu, Jun 9, 2016 at 4:13 PM, Sachith Withana <[email protected]> >>>> wrote: >>>> >>>>> Hi Dulitha, >>>>> >>>>> Can you provide the query that you are using? >>>>> >>>>> That would help us give you a better solution I guess. >>>>> >>>>> Regards, >>>>> Sachith >>>>> >>>>> On Thu, Jun 9, 2016 at 2:34 PM, Dulitha Wijewantha <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi guys, >>>>>> I have written a customer Spark UDF function that allows me to >>>>>> convert a date string to a given input format. If there is a >>>>>> ParseException >>>>>> due to input being invalid - the whole spark script will stop execution. >>>>>> Is >>>>>> there away to skip that particular iteration and move on to the next in >>>>>> the >>>>>> record? (this function is used in where clauses). Or is there a better >>>>>> recommendation for this? Like validation in the EventReceiver? >>>>>> >>>>>> public String dateFormatter(String input, String fromFormat, String >>>>>> toFormat) throws ParseException{ >>>>>> SimpleDateFormat sdf = new SimpleDateFormat(fromFormat); >>>>>> Date date = sdf.parse(input); >>>>>> SimpleDateFormat newFormatter = new SimpleDateFormat(toFormat); >>>>>> return newFormatter.format(date); >>>>>> } >>>>>> >>>>>> Cheers~ >>>>>> >>>>>> -- >>>>>> Dulitha Wijewantha (Chan) >>>>>> Software Engineer - Mobile Development >>>>>> WSO2 Inc >>>>>> Lean.Enterprise.Middleware >>>>>> * ~Email [email protected]* >>>>>> * ~Mobile +94712112165 <%2B94712112165>* >>>>>> * ~Website dulitha.me <http://dulitha.me>* >>>>>> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >>>>>> *~Github @dulichan <https://github.com/dulichan>* >>>>>> *~SO @chan <http://stackoverflow.com/users/813471/chan>* >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Sachith Withana >>>>> Software Engineer; WSO2 Inc.; http://wso2.com >>>>> E-mail: sachith AT wso2.com >>>>> M: +94715518127 >>>>> Linked-In: <http://goog_416592669> >>>>> https://lk.linkedin.com/in/sachithwithana >>>>> >>>> >>>> >>>> >>>> -- >>>> Dulitha Wijewantha (Chan) >>>> Software Engineer - Mobile Development >>>> WSO2 Inc >>>> Lean.Enterprise.Middleware >>>> * ~Email [email protected]* >>>> * ~Mobile +94712112165 <%2B94712112165>* >>>> * ~Website dulitha.me <http://dulitha.me>* >>>> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >>>> *~Github @dulichan <https://github.com/dulichan>* >>>> *~SO @chan <http://stackoverflow.com/users/813471/chan>* >>>> >>> >>> >>> -- >>> Gimantha Bandara >>> Software Engineer >>> WSO2. Inc : http://wso2.com >>> Mobile : +94714961919 >>> >>> >> >> >> -- >> Dulitha Wijewantha (Chan) >> Software Engineer - Mobile Development >> WSO2 Inc >> Lean.Enterprise.Middleware >> * ~Email [email protected] <[email protected]>* >> * ~Mobile +94712112165 <%2B94712112165>* >> * ~Website dulitha.me <http://dulitha.me>* >> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >> *~Github @dulichan <https://github.com/dulichan>* >> *~SO @chan <http://stackoverflow.com/users/813471/chan>* >> > > > > -- > Sachith Withana > Software Engineer; WSO2 Inc.; http://wso2.com > E-mail: sachith AT wso2.com > M: +94715518127 > Linked-In: <http://goog_416592669> > https://lk.linkedin.com/in/sachithwithana > -- Dulitha Wijewantha (Chan) Software Engineer - Mobile Development WSO2 Inc Lean.Enterprise.Middleware * ~Email [email protected] <[email protected]>* * ~Mobile +94712112165* * ~Website dulitha.me <http://dulitha.me>* * ~Twitter @dulitharw <https://twitter.com/dulitharw>* *~Github @dulichan <https://github.com/dulichan>* *~SO @chan <http://stackoverflow.com/users/813471/chan>*
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
