>Do I need to make bulk copy? It depends. If you topology will fail, kafka spout will starts read from last known offset. If you will have too many data to write. And inserting one row can be bottleneck.
You can test it actually, stop topology, write around 10000+/- messages to kafka and start topology. In storm ui you will see capacity for writer bolt. If it red colored and over 1.0 you should notice that and this is your bottleneck. We have kafka to HP Vertica stream. Vertica don't like 1 row inserts and we added batches with 10K rows. With 4 workers everything looks great. On 10 December 2014 at 00:34, Sa Li <[email protected]> wrote: > Hello, all > > I have a question here, as I post several threads before, I am using > storm-rdbms to write into postgresqlDB, data was collected from > kafkaSpout, it works. Since it insert into DB once I get a tuple, per > row/insert operation. I have concern that if this type of consuming is fast > enough and will potentially cost the overhead? > > Do I need to make bulk copy? > > > thanks > > > Alec > -- With best regards, Irek Khasyanov.
