[ https://issues.apache.org/jira/browse/METRON-227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
James Sirota updated METRON-227: -------------------------------- Labels: 0.2.1BETA (was: ) > Add Time-Based Flushing to Writer Bolt > -------------------------------------- > > Key: METRON-227 > URL: https://issues.apache.org/jira/browse/METRON-227 > Project: Metron > Issue Type: Bug > Reporter: Domenic Puzio > Assignee: Ajay Yadav > Labels: 0.2.1BETA > > We need to change the BulkMessageWriterBolt and BulkWriterComponent to use > time-based flushing when writing data to Elasticsearch or Solr. > Currently, we set a batch size, and the Writer waits for that number of > tuples to build up; however, Storm has a timeout value that prevents it from > waiting for too long. If the Writer does not get the batch size before the > timeout, then it recycles the tuples through the topology. In addition, Storm > only allows so many pending messages that have not been acked - if too many > messages are waiting for the bulk Writer, then it will recycle them through > the topology. This is not desired behavior and directly impacts the > performance of this Writer. We would like to be able to specify a unit of > time for which the topology would flush, writing the data it's currently > holding to Elasticsearch or Solr even if the batch size is not met. -- This message was sent by Atlassian JIRA (v6.3.4#6332)