Re: Equivalent of emptyDataFrame in StructuredStreaming

2018-11-05 Thread Arun Manivannan
I also looked into the MemoryStream as a workaround but I do see that it is not recommended for production use. But then, since I am converting this to a DS immediately, I couldn't see a problem with mutation. Could you let me know if this is a better idea than to drop in an empty file (which is e

Equivalent of emptyDataFrame in StructuredStreaming

2018-11-04 Thread Arun Manivannan
Hi, This is going to come off as a silly question with a stream being unbounded but this is problem that I have (created for myself). I am trying to build an ETL pipeline and I have a bunch of stages. val pipelineStages = List( new AddRowKeyStage(EvergreenSchema), new WriteToHBaseStage(hBase