If all sql results have same set of columns you could UNION all the dataframes
Create an empty df and Union all
Then reassign new df to original df before next union all
Not sure if it is a good idea, but it works
Lohith
Sent from my Sony Xperia™ smartphone
---- Divya Gehlot wrote ----
Hi,
Please bear me for asking such a naive question
I have list of conditions (dynamic sqls) sitting in hbase table .
I need to iterate through those dynamic sqls and add the data to dataframes.
As we know dataframes are immutable ,when I try to iterate in for loop as shown
below I get only last dynamic sql result set .
var dffiltered : DataFrame = sqlContext.emptyDataFrame
for ( i <- 0 to (dfFilterSQLs.length - 1)) {
println("Condition============="+dfFilterSQLs(i))
dffiltered =
dfresult.filter(dfFilterSQLs(i)).select("Col1","Col2","Col3","Col4","Col5")
dffiltered.show
}
How can I keep on appending data to dataframe and get the final result having
all the sql conditions.
Thanks in advance for the help.
Thanks,
Divya
Information transmitted by this e-mail is proprietary to Mphasis, its
associated companies and/ or its customers and is intended
for use only by the individual or entity to which it is addressed, and may
contain information that is privileged, confidential or
exempt from disclosure under applicable law. If you are not the intended
recipient or it appears that this mail has been forwarded
to you without proper authority, you are notified that any use or dissemination
of this information in any manner is strictly
prohibited. In such cases, please notify us immediately at
[email protected] and delete this mail from your records.
