Hi folks,
I have DataFrame with let's say this schema :
-dealId,
-ptf,
-ts
from it I derive another dataframe (lets call it df) to which I add an
extra column (withColumn) which is the concatenation of the 3 existing
columns and I call it (the new column) "theone"
When I print the schema for the new dataframe "theone" column has a String
type. And when I do
df.where(df.col("theone").equalTo("nonExistantValue")).toJavaRDD.count well
I get the initial size of df as if the filtring did not work. but If I do
the same query but filtring on one of the original columns I get what is
expected as count which 0
The same goes for isin
Any help will be more than appreciated.
Best regards,
--
Mehdi BEN HAJ ABBES
