Re: Problem in avg function Spark 1.6.3 using spark-shell

2017-06-25 Thread Riccardo Ferrari
Hi, Looks like you performed an aggregation on the ImageWidth column already. The error itself is quite self-explanatory: Cannot resolve column name "ImageWidth" among (MainDomainCode, *avg(length(ImageWidth))*) The column available in that DF are MainDomainCode and avg(length(ImageWidth)) so

Problem in avg function Spark 1.6.3 using spark-shell

2017-06-25 Thread Eko Susilo
Hi, I have a data frame collection called “secondDf” when I tried to perform groupBy and then sum of each column it works perfectly. However when I tried to calculate average of that column it says the column name is not found. The details are as follow val total = secondDf.filter("ImageWidth