Anyway I got it. I have to use !== instead of ===. Thank BTW. On Wed, Dec 9, 2015 at 9:39 PM, Prashant Bhardwaj < prashant2006s...@gmail.com> wrote:
> I have to do opposite of what you're doing. I have to filter non-empty > records. > > On Wed, Dec 9, 2015 at 9:33 PM, Gokula Krishnan D <email2...@gmail.com> > wrote: > >> Hello Prashant - >> >> Can you please try like this : >> >> For the instance, input file name is "student_detail.txt" and >> >> ID,Name,Sex,Age >> =============== >> 101,Alfred,Male,30 >> 102,Benjamin,Male,31 >> 103,Charlie,Female,30 >> 104,Julie,Female,30 >> 105,Maven,Male,30 >> 106,Dexter,Male,30 >> 107,Lundy,Male,32 >> 108,Rita,Female,30 >> 109,Aster,Female,30 >> 110,Harrison,Male,15 >> 111,Rita,,30 >> 112,Aster,,30 >> 113,Harrison,,15 >> 114,Rita,Male,20 >> 115,Aster,,30 >> 116,Harrison,,20 >> >> [image: Inline image 2] >> >> *Output:* >> >> Total No.of Records without SEX 5 >> [111,Rita,,30] >> [112,Aster,,30] >> [113,Harrison,,15] >> [115,Aster,,30] >> [116,Harrison,,20] >> >> Total No.of Records with AGE <=15 2 >> [110,Harrison,Male,15] >> [113,Harrison,,15] >> >> Thanks & Regards, >> Gokula Krishnan* (Gokul)* >> Contact :+1 980-298-1740 >> >> On Wed, Dec 9, 2015 at 8:24 AM, Prashant Bhardwaj < >> prashant2006s...@gmail.com> wrote: >> >>> Already tried it. But getting following error. >>> >>> overloaded method value filter with alternatives: (conditionExpr: >>> String)org.apache.spark.sql.DataFrame <and> (condition: >>> org.apache.spark.sql.Column)org.apache.spark.sql.DataFrame cannot be >>> applied to (Boolean) >>> >>> Also tried: >>> >>> val req_logs_with_dpid = >>> req_logs.filter(req_logs("req_info.dpid").toString.length >>> != 0 ) >>> >>> But getting same error. >>> >>> >>> On Wed, Dec 9, 2015 at 6:45 PM, Fengdong Yu <fengdo...@everstring.com> >>> wrote: >>> >>>> val req_logs_with_dpid = req_logs.filter(req_logs("req_info.pid") != >>>> "" ) >>>> >>>> Azuryy Yu >>>> Sr. Infrastructure Engineer >>>> >>>> cel: 158-0164-9103 >>>> wetchat: azuryy >>>> >>>> >>>> On Wed, Dec 9, 2015 at 7:43 PM, Prashant Bhardwaj < >>>> prashant2006s...@gmail.com> wrote: >>>> >>>>> Hi >>>>> >>>>> I have two columns in my json which can have null, empty and non-empty >>>>> string as value. >>>>> I know how to filter records which have non-null value using following: >>>>> >>>>> val req_logs = sqlContext.read.json(filePath) >>>>> >>>>> val req_logs_with_dpid = req_log.filter("req_info.dpid is not null or >>>>> req_info.dpid_sha1 is not null") >>>>> >>>>> But how to filter if value of column is empty string? >>>>> -- >>>>> Regards >>>>> Prashant >>>>> >>>> >>>> >>> >>> >>> -- >>> Regards >>> Prashant >>> >> >> > > > -- > Regards > Prashant > -- Regards Prashant