Thanks for the response. I read that HDFS is good for only long streaming writes for getting the highest throughput. And latency is huge. In this case for storm, messages are very small. Will this affect the throughput of the system ? Have you seen any other issues with storm-hdfs because of speed mismatch ?
Has storm community suggest something else for the cold storage of the streaming data? On Tue, Jun 28, 2016 at 5:48 PM, Aaron Niskodé-Dossett <[email protected]> wrote: > No, files are not merged. In general you specify a directory and a file > naming convention but each writer writes to its own file. > > On Tue, Jun 28, 2016 at 7:45 PM Jakes John <[email protected]> > wrote: > >> Hi, >> I would like to know how does the storm HDFS bolt works? As far as >> I read about HDFS, it doesn't support multiple writer concurrency on a >> file. ie, Multiple writers cannot write to the same file at the same time. >> Then, how does it work when multiple bolts try to write to a file in the >> HDFS at the same time? I read somewhere that multiple files are created but >> merged at last. Is it true? who is doing the merging? >> >> Thanks, >> Jakes >> >
