What do you mean by workaround? On Tue, Jun 14, 2016 at 9:16 AM, Dulitha Wijewantha <[email protected]> wrote:
> > > On Sat, Jun 11, 2016 at 12:33 AM, Sachith Withana <[email protected]> > wrote: > >> If you need to skip the rows that would be throwing the exception, as you >> said, you'd have to do that in a where clause. >> If there's a scenario where you need to transform the values and use in a >> SELECT statement, use a subquery and filter out the unwanted rows. >> > > Thanks. This can be used as a workaround. But in the future do you plan > to incorporate this as a feature where I can throw a particular exception > to from the UDF that will skip that record or stop the whole script? (Is > this part of the Spark-SQL code too) > > > >> >> ex: >> >> SELECT date FROM ( SELECT dateConvertor(timestamp, "MM/dd/yyyy HH:mm", >> "HOUR") as date FROM eventStatData) Where date != ""; >> >> On Fri, Jun 10, 2016 at 2:25 PM, Dulitha Wijewantha <[email protected]> >> wrote: >> >>> The default value will work for scenarios where we are using the >>> function on a where clause. There is also another scenario where you use it >>> to transform the value in the select. In the below query - if we put a >>> default value it will be returned by the result set rather than ignoring >>> it. >>> >>> select sessionKey as sessionKey, dateFormatter(timestamp, "MM/dd/yyyy >>> HH:mm", "yyyy-MM-dd:HH") as dateDisplay, appKey as appKey, clientModel as >>> role, dateConvertor(timestamp, "MM/dd/yyyy HH:mm", "HOUR") as date from >>> eventStatData >>> >>> On Fri, Jun 10, 2016 at 12:10 AM, Gimantha Bandara <[email protected]> >>> wrote: >>> >>>> You will have to catch the exception inside the method itself and log a >>>> warning/error meesage and return a default value without throwing the >>>> exception out from the method. >>>> >>>> >>>> On Friday, June 10, 2016, Dulitha Wijewantha <[email protected]> wrote: >>>> >>>>> Below is an example usage - >>>>> >>>>> insert overwrite table TABLE_NAME select date, count(distinct >>>>> sessionKey) as activity, appKey, role, first(y.dateDisplay) as dateDisplay >>>>> from (select sessionKey as sessionKey, dateFormatter(timestamp, >>>>> "MM/dd/yyyy HH:mm", "yyyy-MM-dd:HH") as dateDisplay, appKey as appKey, >>>>> clientModel as role, dateConvertor(timestamp, "MM/dd/yyyy HH:mm", "HOUR") >>>>> as date from eventStatData) as y group by y.date, y.appKey, y.role; >>>>> >>>>> >>>>> >>>>> On Thu, Jun 9, 2016 at 4:13 PM, Sachith Withana <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi Dulitha, >>>>>> >>>>>> Can you provide the query that you are using? >>>>>> >>>>>> That would help us give you a better solution I guess. >>>>>> >>>>>> Regards, >>>>>> Sachith >>>>>> >>>>>> On Thu, Jun 9, 2016 at 2:34 PM, Dulitha Wijewantha <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi guys, >>>>>>> I have written a customer Spark UDF function that allows me to >>>>>>> convert a date string to a given input format. If there is a >>>>>>> ParseException >>>>>>> due to input being invalid - the whole spark script will stop >>>>>>> execution. Is >>>>>>> there away to skip that particular iteration and move on to the next in >>>>>>> the >>>>>>> record? (this function is used in where clauses). Or is there a better >>>>>>> recommendation for this? Like validation in the EventReceiver? >>>>>>> >>>>>>> public String dateFormatter(String input, String fromFormat, String >>>>>>> toFormat) throws ParseException{ >>>>>>> SimpleDateFormat sdf = new SimpleDateFormat(fromFormat); >>>>>>> Date date = sdf.parse(input); >>>>>>> SimpleDateFormat newFormatter = new SimpleDateFormat(toFormat); >>>>>>> return newFormatter.format(date); >>>>>>> } >>>>>>> >>>>>>> Cheers~ >>>>>>> >>>>>>> -- >>>>>>> Dulitha Wijewantha (Chan) >>>>>>> Software Engineer - Mobile Development >>>>>>> WSO2 Inc >>>>>>> Lean.Enterprise.Middleware >>>>>>> * ~Email [email protected]* >>>>>>> * ~Mobile +94712112165 <%2B94712112165>* >>>>>>> * ~Website dulitha.me <http://dulitha.me>* >>>>>>> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >>>>>>> *~Github @dulichan <https://github.com/dulichan>* >>>>>>> *~SO @chan <http://stackoverflow.com/users/813471/chan>* >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sachith Withana >>>>>> Software Engineer; WSO2 Inc.; http://wso2.com >>>>>> E-mail: sachith AT wso2.com >>>>>> M: +94715518127 >>>>>> Linked-In: <http://goog_416592669> >>>>>> https://lk.linkedin.com/in/sachithwithana >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Dulitha Wijewantha (Chan) >>>>> Software Engineer - Mobile Development >>>>> WSO2 Inc >>>>> Lean.Enterprise.Middleware >>>>> * ~Email [email protected]* >>>>> * ~Mobile +94712112165 <%2B94712112165>* >>>>> * ~Website dulitha.me <http://dulitha.me>* >>>>> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >>>>> *~Github @dulichan <https://github.com/dulichan>* >>>>> *~SO @chan <http://stackoverflow.com/users/813471/chan>* >>>>> >>>> >>>> >>>> -- >>>> Gimantha Bandara >>>> Software Engineer >>>> WSO2. Inc : http://wso2.com >>>> Mobile : +94714961919 >>>> >>>> >>> >>> >>> -- >>> Dulitha Wijewantha (Chan) >>> Software Engineer - Mobile Development >>> WSO2 Inc >>> Lean.Enterprise.Middleware >>> * ~Email [email protected] <[email protected]>* >>> * ~Mobile +94712112165 <%2B94712112165>* >>> * ~Website dulitha.me <http://dulitha.me>* >>> * ~Twitter @dulitharw <https://twitter.com/dulitharw>* >>> *~Github @dulichan <https://github.com/dulichan>* >>> *~SO @chan <http://stackoverflow.com/users/813471/chan>* >>> >> >> >> >> -- >> Sachith Withana >> Software Engineer; WSO2 Inc.; http://wso2.com >> E-mail: sachith AT wso2.com >> M: +94715518127 >> Linked-In: <http://goog_416592669> >> https://lk.linkedin.com/in/sachithwithana >> > > > > -- > Dulitha Wijewantha (Chan) > Software Engineer - Mobile Development > WSO2 Inc > Lean.Enterprise.Middleware > * ~Email [email protected] <[email protected]>* > * ~Mobile +94712112165 <%2B94712112165>* > * ~Website dulitha.me <http://dulitha.me>* > * ~Twitter @dulitharw <https://twitter.com/dulitharw>* > *~Github @dulichan <https://github.com/dulichan>* > *~SO @chan <http://stackoverflow.com/users/813471/chan>* > -- Sachith Withana Software Engineer; WSO2 Inc.; http://wso2.com E-mail: sachith AT wso2.com M: +94715518127 Linked-In: <http://goog_416592669>https://lk.linkedin.com/in/sachithwithana
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
