Re: Spark - Not contains on Spark dataframe

2017-03-04 Thread KhajaAsmath Mohammed
Hi, I was able to resolve issue with below conditions. datapoint_df(Constants.Datapoint.Vin).like("012345") datapoint_filter_df.filter( datapoint_filter_df(Constants.Datapoint.Vin) rlike "^([A-Z]|[0-9]|[a-z])+$" ) // for checking alpha numeric. Thanks, Asmath On Tue, Feb 28, 2017 at 10:49 AM,

Re: Spark - Not contains on Spark dataframe

2017-02-28 Thread KhajaAsmath Mohammed
Hi, MY dataframe has records with below conditions but dataframe never gets filtered. I am always getting total count of original records even after using below filter function. Am i doing anything wrong here Note: I tied OR and || too def filterDatapointRawCountsDataFrame(datapoint_df:

Spark - Not contains on Spark dataframe

2017-02-28 Thread KhajaAsmath Mohammed
Hi, Could anyone please provide me your suggestions on how to resolve the issue that I am facing with not contains code on dataframe column. Here is the code. My dataframe is not getting filtered with below conditions. I even tried not and ! on Column. any suggestions? def