Re: Reading the most recent text files created by Spark streaming

Yes thanks. I had flume already for twitter so configured it to get data from Kafka source and post it to HDFS. cheers Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw

Re: Reading the most recent text files created by Spark streaming

Hi, An alternative to Spark could be flume to store data from Kafka to HDFS. It provides also some reliability mechanisms and has been explicitly designed for import/export and is tested. Not sure if i would go for spark streaming if the use case is only storing, but I do not have the full pict